Keep adapter passes out of d3's HIP graphs

#1
by subin - opened
vLLM Semantic Router org

With D3_GRAPH_TOKENS > 0, runtime.warmup() captures d3's HIP graphs while every Vela adapter is off. A later pass with an adapter on replays one of those graphs whenever its shape fits, and a graph never reads the adapter gate (vela3_adapters.GATE), so router questions get plain d3-lite answers. On one MI300X with D3_GRAPH_TOKENS=8192, the four router questions moved by up to 0.828 after warm-up against the same requests before it.

With this change, d3's fast path reports no graph shape while an adapter is on, so those passes run eagerly. Passes with every adapter off still replay d3's graphs. Nothing changes at the default D3_GRAPH_TOKENS=0, or off ROCm, where d3's fast path is not installed.

Tested on one MI300X with D3_GRAPH_TOKENS=8192 on 200-token requests: four router questions, router questions mixed with ordinary ones, four ordinary questions, and router questions with a PII span question, each answered before and after runtime.warmup(). Every question with an adapter on now gets the same answer after warm-up as before it, bit for bit. Ordinary questions in a pass of their own still replay d3's graphs and differ from eager by up to 0.0033, as d3-lite does on its own with graphs.

adaamko changed pull request status to merged

Sign up or log in to comment