A Low-Cost Decision Model on CPU: Validating Decisions after Optimizing DiffusionGemma with vLLM - Part II

1. Introduction

In Part I, we made DiffusionGemma’s structured reads faster on CPU by changing how vLLM executes them: vLLM runs every eligible read in two engine steps, and our optimization simplifies it to one. We proposed it upstream in vllm-project/vllm#61480 as a flag, single_pass_reads, which is off by default. On a C4 node with 24 vCPUs, our CPU optimization cut latency at concurrency 1 by 18.8% and the cost of each decision at higher concurrency by 5 to 13%. On a C3 node with 44 vCPUs, it cut them by 21.2% and 7 to 15%, and a run with the flag off showed that the gain comes from single_pass_reads.

Speed and cost were only part of what we needed from a CPU deployment. For CPUs to be a real alternative to GPUs, we set three requirements:

• Keep the model in BF16 with a context of 4,096 tokens.
• Answer a single request in under 1 second at the 95th percentile.
• Keep its decisions stable when the build or the machine changes.

The first requirement is about memory. We wanted to keep the model in its original precision, BF16, with a context that covers the whole read: the prompt plus its canvas. A context of 4,096 tokens leaves room for long prompts. On GPUs, BF16 weights and a long context compete for the same memory: our earlier deployment on two L4 GPUs had to fall back to FP8 and 1,024 tokens. On CPU, there is enough memory for both, which is another reason to explore CPU inference. The second requirement is about speed. We wanted each response on CPU to arrive in a time comparable to what we were used to on GPUs. We found that, with our CPU optimization, both machines tested met the first two requirements, C4 and C3, whose Intel Xeon processors include Advanced Matrix Extensions (AMX).

The third requirement is harder to check than it sounds. The single_pass_reads flag is designed to leave the computation a read performs unchanged and to change only how that work is batched. In BF16, the same computation in a different batch shape can round differently. This means that the probabilities we obtained from DiffusionGemma for a read are close, but not identical. For a decision model, close is good enough only when the decision stays the same. That brings us to the question behind this post: is a faster build still making the same decisions?

This question is not new on GPUs. Thinking Machines showed that the same prompt can produce different outputs depending on the batch it runs in, because common inference kernels are not batch invariant. vLLM v0.32.0, our baseline, offers a batch invariant mode for NVIDIA, AMD and Intel GPUs, but its documentation does not list CPUs. Our setting differs in a second way too. Since we serve a decision model, what matters to us is not whether the tokens match but whether the decisions do. And this is exactly what we set out to measure.

We ended our previous blog post with a first measurement. On JevBench’s 231 public items, our CPU optimization with the flag off changed none of the decisions on C3. With the flag on, it changed 2 on C4 and 4 on C3, and accuracy did not change significantly on either machine. The surprise came when we compared the machines themselves: moving the same vLLM baseline from C3 to C4, with no optimization at all, changed 10 decisions. If the hardware alone moves more decisions than the optimization, what should stable decisions mean?

In the second part of this blog post series, we answer that question with data and benchmarks in hand. We start by explaining why aggregate accuracy hid the changes, and how we compared decisions one item at a time. Next, we use a set of controls to separate the effects of the optimization, batching and hardware, and define what stable decisions should mean. An FP32 run then measures how much of the difference comes from BF16 rounding, not a different computation. Finally, we go back to the three requirements and share the checks we would run again for any change to an inference engine.

2. Accuracy did not change, but decisions did

When we evaluate a change to a model or its serving stack, the first number we usually check is accuracy. For a decision model, accuracy is only part of the answer. We also need to know whether the decisions themselves changed.

To see why, let’s go back to the same example provided in our previous blog post, repeated here for convenience since we are going to refer to it in the following sections as well. This example is illustrative and does not describe a production system.

Consider a hypothetical agent that classifies a member’s message:

I’ve had a headache for 3 days. Is that normal?

The agent asks two questions about it, each with its own options, and sends an answer template with one open slot per question:

Prompt (rubric)  
Topic:    A = billing   B = symptoms   C = scheduling  
Urgency:  A = routine   B = soon       C = urgent

Canvas (answer template)  
Topic:  [ ]  
Urgency:  [ ]

The read returns a probability for every option:

Read output  
Topic:    A 0.01   B 0.97   C 0.02  
Urgency:  A 0.12   B 0.79   C 0.09

The decisions are the options with the highest probability in each question: symptoms for Topic and soon for Urgency. If a new build read the same message and answered routine instead of soon, the user would get a different follow-up, even if the build’s accuracy stayed the same. In healthcare, a decision like this is not a detail. It decides how quickly someone gets the help they asked for.

To measure how often that happens, we needed multiple decisions with their known answers. In Part I, we used JevBench to count engine steps. Here, we use it for what it was built for. JevBench is a public benchmark from Benchmark Heaven for decision models like Jev. Each item gives the model a piece of state and a bounded set of options, and asks for a probability for every option. In other words, every JevBench item is a structured read like the one above, with a single question. As our previous post showed, on the C3 VM all 231 public items ran in one engine step each with the flag on.

Although JevBench is not healthcare data, its items are the same kind of read as ours: a piece of state, a bounded set of options and a probability for each. What changes a decision does not depend on the domain, but how many decisions change depends on how close a workload’s items sit to a decision boundary.

Three properties make JevBench a good fit for measuring decision changes:

• Every item has a gold label. Each item comes with the option its authors judged correct. When a decision changes, we can tell whether it became correct or incorrect. Unlike our own synthetic workload from Part I, these labels are public and independent of us, and anyone can check them.
• The items are fixed. We pinned JevBench at revision 1bcc55e, and every run reads the same 231 public items. That lets us pair each item with itself across builds and machines.
• The items come in three tiers. 48 are easy, 72 are standard and 111 are hard. The hard items were written to be difficult, then reviewed and frozen before any system ran. The tiers show where a change lands.

With a gold label for every item, the natural first comparison is accuracy: the share of the 231 items where the decision matches the gold label. By that measure, the CPU optimization matches the baseline, vLLM v0.32.0. On C4, with reads sent sequentially, it answered 194 items correctly against 196 for the baseline, a drop of 0.87 percentage points. On C3, it went the other way: our CPU optimization answered 196 items correctly, a gain of 0.87 percentage points.

Table 1. JevBench accuracy and changed decisions. The CPU optimization build with single_pass_reads on against the baseline with vLLM v0.32.0 shows no statistically significant changes in accuracy when running on C3 or C4 machines.

Differences that small can come from chance. To check, we used an exact McNemar test, a standard test for comparing two models on the same items. It ignores the items that both builds got right or both got wrong, and looks only at the items where one build was correct and the other was not. We found that neither change was statistically significant, on C4 or on C3 (Table 1).

So far, accuracy says nothing has changed. The last three columns of Table 1 tell us something else: our CPU optimization changed 2 decisions on C4 and 4 on C3, or 0.9% and 1.7% of the 231 items. In our runs, the accuracy numbers missed these changes for three reasons:

• Changes in opposite directions cancel out. On C3, 3 decisions became correct and 1 became incorrect. Accuracy only shows what is left after they cancel: a net gain of 2 items.
• Changes between two wrong options are invisible. When we sent 4 concurrent reads on C3, one decision moved from one wrong option to another, and accuracy did not change at all. In our example, that would be like Urgency moving from routine to urgent when the right answer is soon. Accuracy counts both answers as wrong, but the user would get a very different follow-up.
• A few changes are too few for the test to flag. The McNemar test needs at least 6 items where one build was right and the other was not before it can report a significant difference. On C4, both changed decisions became incorrect, and the test still could not flag them. With 2 such items on C4 and 4 on C3, it could not have found a difference, whatever the decisions did.

So, should we stop caring about accuracy?
No. We do care about accuracy, but we must check changed decisions too. They answer different questions, and in healthcare both matter. Accuracy asks whether our CPU optimization is as good as the baseline. Changed decisions ask whether the same message still gets the same answer.

As the earlier example shows, a different follow-up for the same message matters to the user, whatever the aggregate accuracy says. That is why we cannot report accuracy alone, and why we pinned the exact JevBench revision we used, so that anyone can rerun this test. From here on, we report changed decisions and their direction alongside accuracy.

In the next section, we describe how we compared decisions item by item, and the controls we used to answer it.

3. How we compared decisions

To judge the changes our CPU optimization made, we need two things: a way to compare two reads of the same item, and a set of controls that change one thing at a time. This section describes both.

3.1 Comparing two reads of the same item

As we saw in Section 2, JevBench has 231 items, each with a single question. That lets us compare our CPU optimization with the baseline one item at a time. For each item, we see how the baseline reads it, and compare with the one from our CPU optimization. We repeat this for all items.

Every other comparison in this post, including the controls in Section 3.2, is performed the same way (Figure 1). For each pair of reads, we record three things:

• Whether the decision changed. The option with the highest probability differs between the two reads.
• Its direction. Against the gold label, a changed decision becomes correct, becomes incorrect or moves between two wrong options.
• How far the probabilities moved. We measure this shift with the Total Variation (TV) distance between the two reads, expressed in percentage points (pp).

Figure 1. How we compare two reads of the same item. The same item is read under two setups, the baseline and our CPU optimization. We record whether the decision changed, its direction against the gold label, and how far the probabilities moved (TV). The read is an illustrative example, not a JevBench item.

TV deserves a closer look, because the rest of this post relies on it. It is half the sum of the absolute differences between the two reads’ probabilities, and we report it in percentage points. Two identical reads have a TV of 0, and two reads that put all their probability on different options have a TV of 100 pp.

Using our example once again, let’s assume we read the same message a second time. The comparison we perform then takes the baseline’s read of the message and our CPU optimization’s read of the same message, and compares the two Urgency answers:

Urgency:  A = routine   B = soon    C = urgent

Urgency  
Baseline:   A 0.12   B 0.79   C 0.09  
CPU opt:    A 0.15   B 0.74   C 0.11

The decision is still soon, and a count of changed decisions would not show any difference. The TV between the two reads, however, is 5 pp: half of 0.03 + 0.05 + 0.02.

In summary, TV tells us how far a read moved, even when its decision did not. Over the 231 items, each comparison then reports the number of changed decisions, their direction, and the mean and 95th percentile of the TV.

3.2 Changing one thing at a time

A changed decision tells us that something moved, not what moved it. Between the baseline on C3 and our CPU optimization on C4, three things differ at once: the build, the machine and, at higher concurrency, how reads are batched together. To separate them, we used four comparisons, each of which changes one thing and keeps the rest fixed:

• Repeat. The baseline against itself, read twice on the same endpoint. Nothing changes between the two reads, and any difference should be attributed to noise from the serving stack.
• Batching. The same build, with reads sent sequentially and then at concurrency 4, meaning four concurrent requests. Concurrent reads share engine steps, and as Part I noted, the same computation in a different batch shape can round differently in BF16.
• Optimization. The CPU optimization against the baseline, on the same machine and at the same concurrency. On C3, we also ran the CPU optimization with the flag off. That build carries the two prompt-logprob changes but runs reads on the original path, and it separates single_pass_reads from them.
• Hardware. The same build on C3 and on C4.

We ran these comparisons on the Vertex AI endpoints from Part I, on C4 (c4-standard-24) and C3 (c3-highcpu-44). Each build read all 231 items sequentially, and then again at concurrency 4. We then repeated the whole run with the builds in reverse order, to check whether the order in which they ran mattered. We found that the order did not affect the results. With sequential reads, every build returned the same probabilities in both passes, down to the last bit, and every comparison in Figure 2 gives the same number of changed decisions in either order. Since both passes agree, the rest of this post reports only the first one.

On C3, we also ran the same comparisons on the VM from Part I (c3-standard-44). There, unlike on Vertex AI, we could reset the prefix cache between builds and count engine steps.

4. Results: Where the changes come from

Whether our CPU optimization meets our third requirement comes down to its decisions. Do they stay the same as the baseline’s, and if they change, how many change and why? Before we can answer that, we need to know whether decisions change when we read the same build twice. We start there, because every other result in this post is measured against it.

Reading the baseline twice, we found that:

• With sequential reads, sent one at a time, nothing changes at all. All 231 items came back with identical probabilities, down to the last bit, on both machines. A build that does nothing different changes none of its decisions. This is what we expected: with nothing else in the batch, the same read goes through the same computation every time.
• With four concurrent reads, results vary a little. The server groups requests that arrive close together into the same engine step, and the timing differs from run to run. A different group means a different batch shape, and in BF16 a slightly different result. Reading the baseline twice changed 0 decisions on C3 and 1 on C4.

This gives us the rule we use for the rest of this post. Zero changed decisions is the right reference for repeatability, but it is not a fair bar for a change to the engine. Serving the baseline at concurrency 4 already changes some decisions, and any deployment that batches requests already accepts that. We therefore call a change’s decisions stable when it changes at most one more decision than batching the baseline does, and when a higher precision shows no evidence of a different computation. We set this rule after our first runs, not before them, and we report every comparison so readers can apply a stricter one.

Reads sent sequentially are our reference point: zero changed decisions on both machines. They are the only setting where nothing at all varies between two runs.

Now we can go back to the changes from Section 2 and ask what caused them. Figure 2 shows every comparison: on C4 in panel (a), on C3 in panel (b), and between the two machines in panel (c). Most rows use reads sent sequentially. The batching rows compare sequential reads with concurrency 4. The rows marked concurrency 4 compare two runs that were both served at concurrency 4.

Figure 2. Changed decisions on JevBench’s 231 public items. Every changed decision is on the hard tier. (a) On C4. (b) On C3. (c) The same build, moving from C3 to C4. Batching goes from reads sent sequentially to 4 concurrently. Rows marked concurrency 4 read both sides at concurrency 4.

We started with our CPU optimization against the baseline. With the flag on, every one of the 231 items came back with probabilities different from the baseline’s, on both machines. The mean TV (the total variation distance we introduced in Section 3) was 2.30 pp on C3 and 1.83 pp on C4. In other words, our CPU optimization nudged every item’s probabilities by about 2 points on average, but almost none of those nudges were big enough to change an answer. Only 4 decisions changed on C3 (Figure 2b) and 2 on C4 (Figure 2a).

Every changed decision in Figure 2 is on JevBench’s hard tier. No decision on an easy or standard item changed in any comparison. This follows from how close each call is. On easy and standard items, the top option is usually ahead of the second by almost 100 pp, with a median gap of 99.7 pp. Even the closest of these 120 items was 32 pp ahead, and our CPU optimization moved none of them by more than 4.2 pp. To swap the top two options, a read has to move by at least half the gap between them. On hard items, 28 of 111 had a gap under 50 pp, and they also moved the most, by up to 43 pp on C3 and 38 pp on C4.

Compared with batching and hardware, the changes from our CPU optimization are not unusual. On C4, our CPU optimization changed 2 decisions, and both became incorrect. Sending the baseline’s reads at concurrency 4 changed the same number, with the same direction (Figure 2a). On C3, our CPU optimization changed 4 decisions and batching changed 3 (Figure 2b). Moving the same baseline from C3 to C4, with no optimization at all, changed 10 decisions, more than our CPU optimization or batching the baseline on either machine. Moving the build with our CPU optimization from C3 to C4 changed 6, against 10 for the baseline, with almost the same probability shift (Figure 2c and Table 2).

The probability shifts give a clearer picture than the counts, because they average over all 231 items (Table 2). On both machines, our CPU optimization moved probabilities slightly less than batching the baseline did, and switching machines moved them the most on average. With the flag off, our CPU optimization changed none of the decisions on C3 (Figure 2b), so on C3, every change comes from single_pass_reads itself.

Table 2. Probability shifts on JevBench’s 231 public items, in pp.

At concurrency 4, our CPU optimization does not behave like the baseline in two ways. On C3, sending its reads 4 concurrently instead of sequentially changed 10 decisions, against 3 for the baseline. Six of the 10 became incorrect. The Vertex endpoint and the VM both show 10. On C4, batching changed 2 decisions for both builds. We also compared the two builds directly, with both at concurrency 4. Our CPU optimization changed 4 decisions on C4, 3 of which became incorrect, and 7 on C3, 4 of which became incorrect (Figure 2a and 2b). Reading the baseline twice at concurrency 4 changed 1 decision on C4 and none on C3. At concurrency 4, our CPU optimization changes more decisions than timing alone does on either machine. All of these items are among the 23 we reran in FP32 in Section 5, and none of their decisions changed there. We do not yet know why batching affects our CPU optimization more on C3 than on C4.

So far, we have counted where decisions change. In the next section, we try to answer why decisions change.

5. Why decisions change: BF16 rounding

At the start of this post, we claimed that single_pass_reads does not change the computation a read performs, only how that work is batched. In BF16, the original precision of DiffusionGemma, the same computation in a different batch shape can round differently. If our claim is true, every changed decision in Section 4 comes from rounding.

To verify our claim, we need a way to discriminate the effect of rounding from a bug, and a higher precision does exactly that. With the flag on and off, the two builds batch the same computation differently, in BF16 and in FP32 alike. What changes in FP32 is the size of the rounding error, which becomes much smaller. If the two builds make the same decisions in FP32, rounding was the cause. If they still disagree, our CPU optimization computes something different, and that points to a bug. In our example, if Urgency moved from soon to routine in BF16 but stayed soon in FP32, the cause of the change would lie in BF16 rounding, not in the computation itself.

To discriminate this effect, we reran the 23 most sensitive JevBench items on the C3 VM, in FP32 and in BF16, with the flag on and off. These 23 selected items are the ones whose decisions changed in our earlier runs, plus the ones whose probabilities moved the most. Each build read them sequentially and then at concurrency 4.

We found that in FP32, the flag changed none of the 23 decisions, sequentially or at concurrency 4. With sequential reads, the mean TV was 0.07 pp, and no item moved by more than 1 pp. In BF16, on the same items, the flag changed 4 decisions with a mean TV of 14.6 pp (Figure 3). That TV is much larger than the 2.30 pp of Section 4, because these items were chosen for being the most sensitive. With rounding error this small, our CPU optimization makes the same decisions with the flag on and off on these items.

Figure 3. Probability shift from our CPU optimization, in BF16 and in FP32. Each row is one of the 23 most sensitive JevBench items, read sequentially on the C3 VM. Bars show the shift in BF16, with the 4 items whose decision changed in a darker shade. Black dots show the shift in FP32, under 1 pp for every item.

Table 3. Changed decisions and probability shifts on the 23 most sensitive JevBench items, on the C3 VM.

The BF16 batching counts here are higher than in Section 4. This run used vLLM main, and these long prompts are split across engine steps differently at concurrency 4. The comparison that matters is BF16 against FP32 within this run.

FP32 also removed the batching effect on decisions. Sending reads at concurrency 4 changed none of the 23 decisions in FP32, for either build (Table 3). The 23 items include all 10 that batching changed for our CPU optimization on C3 in Section 4. This points to rounding as well, with one exception in the probabilities that we describe below.

The last comparison changes nothing but the precision. On the original path, with the flag off, moving from FP32 to BF16 changed 5 of the 23 decisions, 2 became correct and 3 became incorrect, with a mean TV of 17.2 pp. Our CPU optimization changed 4, with a mean TV of 14.6 pp. On the selected items, choosing BF16 over FP32 moves about as many decisions as our CPU optimization does, and moves probabilities about as far.

Before we draw conclusions, there are four things to keep in mind about this test:

• The 23 items are the most sensitive ones. They were chosen because they changed or moved the most, and they say nothing about how often such items occur.
• FP32 is used as a diagnostic. It doubles the memory the weights need, and each read took about four times longer: a median of 7.3 s against 1.9 s in BF16, with reads sent sequentially. Both are slow because these 23 items have long prompts, with a median of 2,355 tokens against 185 for all 231 items.
• The FP32 run used vllm-project/vllm#61480 on vLLM main, not on v0.32.0 like the rest of this post. The single pass is the same change in both.
• One result is still open. One item with a long prompt (3,493 tokens) shifted by 9.4 pp in FP32, but only with the flag on at concurrency 4. With the flag off, the same item moved by 0.2 pp, and every other item moved by less than 1 pp. Its decision did not change, and we are still looking into it.

Overall, the results obtained with FP32 point to why decisions change: the changed decisions are consistent with BF16 rounding, and we found no evidence of a different computation. In the last section, we bring these results together and go back to the question we started with.

6. Conclusions: what stable decisions mean

We started this post with a question: is a faster build still making the same decisions? Accuracy alone could not answer it. On JevBench, accuracy did not change significantly on either machine, but a few decisions did. To find where they came from, we read each of the 231 items under two setups and compared the two reads of the same item, changing one thing at a time: the optimization (our CPU optimization against the baseline), the batching (sequential reads against concurrency 4), the hardware (C3 against C4) and the precision (BF16 against FP32, on the 23 most sensitive items).

Our rationale for changing the precision comes from how our CPU optimization is designed: it should not change the computation a read performs, only how that work is batched. In BF16, a different batch shape can round differently, and that alone can change a decision. In FP32, our CPU optimization still batches the work differently with the flag on and off, but the rounding error becomes much smaller. If rounding is the only reason decisions change, the changes should disappear in FP32. That is what the precision comparison tested.

We found that in BF16, our CPU optimization is not the only thing that changes decisions. With the same baseline, serving reads at concurrency 4 or moving from C3 to C4 already changes some of them. What matters is how many decisions change, where they change and why.

We can now answer the three key questions that tell us whether our CPU optimization keeps decisions stable:

• How many decisions change? A few. With sequential reads, our CPU optimization changed 2 of 231 decisions on C4 and 4 on C3 (0.9% and 1.7%), at most one more than batching the baseline (2 and 3), and fewer than moving it from C3 to C4 (10). At concurrency 4, batching differs by machine. On C4, batching changed 2 decisions for both builds. On C3, it changed 10 for our CPU optimization, 6 of which became incorrect, against 3 for the baseline. Compared directly at concurrency 4, our CPU optimization changed 4 decisions on C4 and 7 on C3, against 1 and 0 when the baseline was read twice.
• Where do they change? Every changed decision was on JevBench’s hard tier, where the model already struggles. No easy or standard item changed in any comparison.
• Why do they change? With the flag off, no decision changed on C3, so the changes come from single_pass_reads itself. In FP32, neither the flag nor batching changed a decision on any of the 23 most sensitive items. We found no evidence of a different computation, and the changes are consistent with BF16 rounding.

In other words, our CPU optimization changes only a few decisions, at most one more than batching already does, only on items in the hard tier, and with no evidence of a different computation. This is the case on both machines with sequential reads.

Our results offer the answer to the third requirement. In the first part of this blog series, we found that our CPU optimization met the first two on both C3 and C4. In this second part, we asked what stable decisions should mean. Under the rule from Section 4, decisions are stable when a change moves at most one more decision than batching the baseline does, and shows no evidence of a different computation. Under this definition, both machines meet the third requirement with sequential reads. At concurrency 4, neither does yet. On C4, batching changes as many decisions for our CPU optimization as for the baseline, but the two builds compared directly differ on 4 items, against 2 when the baseline’s reads are batched and 1 when it is read twice. On C3, batching changed 10 decisions for our CPU optimization, against 3 for the baseline.

When the machine changes, our CPU optimization moves no more decisions than the baseline does: 6 from C3 to C4, against 10. With sequential reads, C4 meets all three requirements and costs the least to keep running. We identified it as the best candidate for decision workloads whose requests mostly arrive one at a time. For those, a single C4 replica costs, as of November 2026, about a fifth of an A100 and 40% less than two L4 GPUs to keep running, which makes CPUs a real alternative to GPUs. Bursts are the limit. In Part I, p95 latency on C4 rose above 1 second at concurrency 3, and at concurrency 4 our CPU optimization does not yet meet our rule for stable decisions.

Looking back, these are the checks we would run again for any change to an inference engine, not only ours:

• Read the baseline twice first. We need to know how much a build varies on its own before any other comparison means anything.
• Separate the change from batching and hardware. Both change decisions on their own. Without these controls, every change gets blamed on the build.
• Count changed decisions, not only accuracy. Accuracy can stay flat while decisions move in both directions. We report their direction and how far the probabilities moved.
• Use a higher precision to tell rounding from a bug. If a change disappears in FP32, it points to rounding. If it survives, it points to a bug.

single_pass_reads is under review upstream in vllm-project/vllm#61480, and it is off by default. If you serve DiffusionGemma’s structured reads on CPU, give it a try, and run these checks on your own workload before you turn it on.

Try it out!

single_pass_reads is under review upstream in vllm-project/vllm#61480, and it is off by default. If you serve DiffusionGemma’s structured reads on CPU, give it a try, and run these checks on your own workload before you turn it on. With that build, the serving settings from this post are:

vllm serve google/diffusiongemma-26B-A4B-it \
    --dtype bfloat16 --max-model-len 4096 \
    --max-num-seqs 16 --max-num-batched-tokens 8192 \
    --max-logprobs 32 --enable-prefix-caching \
    --diffusion-config '{"canvas_length": 64, "single_pass_reads": true}'

Structured reads come from vLLM’s example decision server, which sends one-step reads by default.

Acknowledgements

Thanks to the vLLM reviewers who helped land the fixes that made structured reads run on CPU. The changes behind our CPU optimization are still under review vllm-project/vllm#61480.

Thanks also to Benchmark Heaven for JevBench, a public benchmark with frozen items and gold labels, which made every comparison in this post reproducible.

Special thanks to Maciej Kozlowski and Aiden Arnold for peer reviewing both parts of this series, and to Mike Doane for the editorial review at League.