# Continuous batching

Continuous batching maximizes GPU utilization by dynamically rescheduling the batch at every generation step. As requests finish, new ones join immediately instead of waiting for the whole batch to complete. The GPU stays full and throughput stays high.

> [!TIP]
> For production deployments, use [transformers serve](./serve-cli/serving_optims#continuous-batching). It builds on [ContinuousBatchingManager](/docs/transformers/v5.21.0/en/main_classes/continuous_batching#transformers.ContinuousBatchingManager) and exposes an OpenAI-compatible HTTP endpoint.

## generate_batch

Continuous batching is supported through [generate_batch()](/docs/transformers/v5.21.0/en/main_classes/continuous_batching#transformers.ContinuousMixin.generate_batch). Pass a list of tokenized prompts and get back results for all of them when they're done. `generate_batch` handles scheduling internally and blocks until all requests are complete.

For serving and streaming use cases, use [ContinuousBatchingManager](#continuousbatchingmanager) directly to manage requests.

```py
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from transformers.generation import ContinuousBatchingConfig, GenerationConfig

model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen3-4B",
    attn_implementation="flash_attention_2",
    device_map="auto",
    dtype=torch.bfloat16,
)
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3-4B")

prompts = [
    "What's up?",
    "Name a cat breed.",
    "Write a detailed history of quantum mechanics.",
]
inputs = [tokenizer.encode(p) for p in prompts]

generation_config = GenerationConfig(
    max_new_tokens=64,
    eos_token_id=tokenizer.eos_token_id,
)

outputs = model.generate_batch(inputs=inputs, generation_config=generation_config)

for request_id, output in outputs.items():
    text = tokenizer.decode(output.generated_tokens, skip_special_tokens=True)
    print(f"[{request_id}] {text}")
```

## ContinuousBatchingManager

[ContinuousBatchingManager](/docs/transformers/v5.21.0/en/main_classes/continuous_batching#transformers.ContinuousBatchingManager) runs a background thread and lets you submit requests and retrieve results independently. Every generation step, it checks for finished requests and schedules new ones to join the batch. This is useful for streaming, real-time serving, or submitting requests as they arrive.

Use [continuous_batching_context_manager()](/docs/transformers/v5.21.0/en/main_classes/text_generation#transformers.ContinuousMixin.continuous_batching_context_manager) to start and stop the manager safely. The example below contains variable length inputs. As soon as the shortest prompt is complete, it leaves the batch while the longer prompts continue generating. With static batching, you'd have to pad them all to the same length. Continuous batching frees up the completed prompt so you can start processing the next prompt immediately.

```py
with model.continuous_batching_context_manager(generation_config=generation_config) as manager:
    manager.add_request(
        input_ids=tokenizer.encode("Write a detailed history of quantum mechanics."),
        request_id="long",
        max_new_tokens=512,
    )
    manager.add_request(
        input_ids=tokenizer.encode("What's up?"),
        request_id="short_0",
        max_new_tokens=32,
    )
    manager.add_request(
        input_ids=tokenizer.encode("Name a cat breed."),
        request_id="short_1",
        max_new_tokens=32,
    )

    for result in manager:
        text = tokenizer.decode(result.generated_tokens, skip_special_tokens=True)
        print(f"[{result.request_id}] {text}")
```

You could also call [init_continuous_batching()](/docs/transformers/v5.21.0/en/main_classes/text_generation#transformers.ContinuousMixin.init_continuous_batching) to manage the lifecycle yourself.

```py
manager = model.init_continuous_batching(generation_config=generation_config)
manager.start()

# submit and retrieve requests...
```

### Shutting down the manager

The manager runs a background thread and holds distributed resources. Shutdown happens in two stages so you can choose what to do with in-flight work.

Call [stop()](/docs/transformers/v5.21.0/en/main_classes/continuous_batching#transformers.ContinuousBatchingManager.stop) to halt the background thread. By default, the manager stops accepting new submissions and waits for queued and active requests to finish before the thread exits.

```py
manager.stop()
```

Pass `hard_stop=True` to abandon pending work immediately. Queued and active requests are failed with a `RuntimeError` instead of finishing.

```py
manager.stop(hard_stop=True)
```

Once `stop` is called, [add_request()](/docs/transformers/v5.21.0/en/main_classes/continuous_batching#transformers.ContinuousBatchingManager.add_request) and [add_requests()](/docs/transformers/v5.21.0/en/main_classes/continuous_batching#transformers.ContinuousBatchingManager.add_requests) drop new submissions and log a warning. You can still call `start` again to run another generation session with the same manager.

Call [destroy()](/docs/transformers/v5.21.0/en/main_classes/continuous_batching#transformers.ContinuousBatchingManager.destroy) to release distributed resources. `destroy` stops the manager first if it's still running, and the manager cannot be restarted afterwards. Use it when you're done with continuous batching for the lifetime of the process.

```py
manager.destroy()
```

[continuous_batching_context_manager()](/docs/transformers/v5.21.0/en/main_classes/text_generation#transformers.ContinuousMixin.continuous_batching_context_manager) handles this process. It calls `stop` on exit and `destroy` unless you pass `persistent_manager=True` to cache the manager on the model for the next session.

### Adding requests

[add_request()](/docs/transformers/v5.21.0/en/main_classes/continuous_batching#transformers.ContinuousBatchingManager.add_request) submits a single request. Provide a `request_id` or let the manager generate one automatically.

```py
manager.add_request(input_ids=input_ids, request_id="my_request")
```

[add_requests()](/docs/transformers/v5.21.0/en/main_classes/continuous_batching#transformers.ContinuousBatchingManager.add_requests) submits a batch at once. It sorts inputs automatically to maximize prefix cache hits when block sharing is enabled.

```py
manager.add_requests(inputs=inputs)
```

Cancel a request with [cancel_request()](/docs/transformers/v5.21.0/en/main_classes/continuous_batching#transformers.ContinuousBatchingManager.cancel_request).

```py
manager.cancel_request(request_id="my_request")
```

### Per-request sampling parameters

Enable `per_request_processors` to apply `temperature`, `top_k`, and `top_p` independently per request within the same forward pass to allow different sampling parameters for different requests (creative, high-temperature outputs versus precise, low-temperature ones for example).

```py
cb_config = ContinuousBatchingConfig(per_request_processors=True)

# each request gets its own sampling parameters
manager.add_request(input_ids=inputs_a, temperature=0.9, top_p=0.95)
manager.add_request(input_ids=inputs_b, temperature=0.1, top_k=10)
```

Each parameter in [GenerationConfig](/docs/transformers/v5.21.0/en/main_classes/text_generation#transformers.GenerationConfig) must be a non-default value in order to create the associated logits processor at runtime. For example, set `temperature` to a value other than `None` or `1` to support per-request temperature control. Requests with temperatures of `1` can still be created afterwards.

### Retrieving results

Iterate over the manager to receive results as they arrive.

```py
for result in manager:
    print(tokenizer.decode(result.generated_tokens, skip_special_tokens=True))
```

[get_result()](/docs/transformers/v5.21.0/en/main_classes/continuous_batching#transformers.ContinuousBatchingManager.get_result) fetches the next result from the output queue. Pass a `request_id` to filter for a specific request. If the next result in the queue doesn't match, it's requeued and the method returns `None`.

```py
# next available result
result = manager.get_result()

# filter for a specific request
result = manager.get_result(request_id="my_request")
```

### Streaming

Set `streaming=True` on a request, then use [request_id_iter()](/docs/transformers/v5.21.0/en/main_classes/continuous_batching#transformers.ContinuousBatchingManager.request_id_iter) to iterate over partial outputs as tokens are generated.

```py
from transformers.generation.continuous_batching import RequestStatus

manager.add_request(input_ids=input_ids, request_id="streamed", streaming=True)

for chunk in manager.request_id_iter(request_id="streamed"):
    token = tokenizer.decode(chunk.generated_tokens[-1:], skip_special_tokens=True)
    print(token, end="", flush=True)
    if chunk.status == RequestStatus.FINISHED:
        break
```

## ContinuousBatchingConfig

[ContinuousBatchingConfig](/docs/transformers/v5.21.0/en/main_classes/continuous_batching#transformers.ContinuousBatchingConfig) controls the KV cache, scheduling, CUDA graphs, memory usage, and more. Pass it to `generate_batch` or [init_continuous_batching()](/docs/transformers/v5.21.0/en/main_classes/text_generation#transformers.ContinuousMixin.init_continuous_batching) with the `continuous_batching_config` argument, separately from [GenerationConfig](/docs/transformers/v5.21.0/en/main_classes/text_generation#transformers.GenerationConfig). The two configs describe different things. [GenerationConfig](/docs/transformers/v5.21.0/en/main_classes/text_generation#transformers.GenerationConfig) is model-centered and holds sampling and stopping parameters, while [ContinuousBatchingConfig](/docs/transformers/v5.21.0/en/main_classes/continuous_batching#transformers.ContinuousBatchingConfig) is hardware-centered and holds memory and scheduling parameters.

> [!WARNING]
> Setting `continuous_batching_config` on a [GenerationConfig](/docs/transformers/v5.21.0/en/main_classes/text_generation#transformers.GenerationConfig) is deprecated, emits a `FutureWarning`, and will be removed in v5.19.

By default, `max_batch_tokens` is `8192`, bounded by available GPU memory and never below `256`, while `num_blocks` fills the remaining memory. Use the table below to help you pick the appropriate features.

| Feature | Memory | Throughput | Latency |
|---|---|---|---|
| `max_memory_percent` / `page_size` | ✓ controls KV budget | | |
| `max_batch_tokens` | ↑ larger input buffers | ✓ bigger prefill batches | ✓ TTFT when prefill-bound |
| `scheduler` | | ✓ scheduling policy | ✓ TTFT |
| CUDA graphs | ↑ graph storage | ✓ less dispatch overhead | ✓ |
| Async batching | ↑ ~2× I/O buffers | ✓ overlaps CPU/GPU | |
| Compilation | ↑ warmup-time only | ✓ faster forward passes | ✓ |
| Decode fast path | ↑ block table per request | ✓ faster decode-only steps | ✓ |
| CPU offloading | ↑ pinned CPU memory | ✓ skips some re-prefills | |
| Prefix caching | ↓ shared KV blocks | ✓ skips redundant prefill | ✓ TTFT |
| Paged attention | ↓ no fragmentation | ✓ dynamic batch membership | |
| Sliding window | ↓ bounded KV per layer | | |
| Per-request processors | | ✓ mixed sampling params per batch | |
| `max_requests_per_batch` | ↓ caps logits buffer | ✓ bounds batch size | |
| `safety_margin` | | | ✓ protects decode latency |

```py
from transformers.generation import ContinuousBatchingConfig

cb_config = ContinuousBatchingConfig(
    max_memory_percent=0.8,
    page_size=256,           # KV cache page size in tokens
    scheduler_type="fifo",        # "fifo" or "prefill_first"
)

outputs = model.generate_batch(
    inputs=inputs,
    generation_config=generation_config,
    continuous_batching_config=cb_config,
)
```

### KV cache page size

`page_size` sets how many tokens of KV a full attention page holds for one layer. It must be at least `4`, or initialization raises a `ValueError`. The default `256` works for most workloads.

> [!NOTE]
> The `block_size` parameter is deprecated and replaced by `page_size`.

A larger page size cuts bookkeeping but wastes space when a sequence stops mid-page. Smaller pages waste less space but need more bookkeeping. You usually don't need to tune `page_size`. If your workload has many short requests, try a smaller value. If it has a few very long ones, try a larger value. See [Memory management](./continuous_batching_architecture#memory-management) for how pages, blocks, and chunks fit together.

```py
cb_config = ContinuousBatchingConfig(page_size=128)
```

### Prefill batch size

`max_batch_tokens` sets the token budget for a single forward pass, the maximum number of query tokens the model processes at once. A larger budget packs more prompt tokens into each prefill, which raises prefill throughput and lowers time-to-first-token on prompt-heavy workloads. The scheduler splits prompts that exceed the budget across steps. See [chunked prefill](./continuous_batching_architecture#chunked-prefill) for how oversized prompts are handled.

By default, `max_batch_tokens` is `8192`. The value is bounded by available GPU memory so the per-batch input tensors don't crowd out the KV cache, and it never falls below `256`. On a GPU with little free memory, the bound lowers it below `8192`.

`max_batch_tokens` and `num_blocks` draw from the same memory budget, so they trade off against each other. A larger token budget allocates bigger input buffers and leaves fewer KV cache blocks for concurrent or long requests. Raise `max_batch_tokens` to push prefill throughput when you have more memory available, and lower it to free memory for more KV blocks or to avoid out-of-memory errors.

```py
cb_config = ContinuousBatchingConfig(max_batch_tokens=16384)
```

### Batch and scheduling limits

`max_requests_per_batch` caps how many requests run in a single forward pass. The model produces a logits tensor sized by the number of batch tokens and the vocabulary size, and it's cast to `fp32` for sampling, which doubles the footprint. Each request only needs one next-token prediction, so a smaller cap keeps this tensor smaller and avoids out-of-memory errors on prefill-heavy batches with large vocabularies. By default it's set to the number of prompts submitted, with a fallback of 1024, then capped to fit `max_batch_tokens` and `num_blocks`.

```py
cb_config = ContinuousBatchingConfig(max_requests_per_batch=256)
```

`safety_margin` reserves a fraction of free cache capacity for active requests. When free capacity falls below `safety_margin` (a fraction of total cache size), the scheduler stops admitting new prefills and only continues decoding requests already in progress. This prioritizes finishing active work over starting new requests, which protects decode latency and delays [offloading](./continuous_batching_architecture#offloading). The value must fall between `0` and `1`, where `0` disables the margin.

```py
cb_config = ContinuousBatchingConfig(safety_margin=0.15)
```

The default depends on the scheduler. For the FIFO scheduler, it's `0.15` and for `prefill_first`, it's `0.0`. See [Admission](./continuous_batching_architecture#admission) for how the margin interacts with scheduling.

### Log probabilities

[ContinuousBatchingConfig](/docs/transformers/v5.21.0/en/main_classes/continuous_batching#transformers.ContinuousBatchingConfig) returns each generated token's log probability when `return_logprobs=True`. This is useful for RL where logprobs are an input to some of the training loops.

```py
cb_config = ContinuousBatchingConfig(return_logprobs=True)

# generate_batch()

for request_id, output in outputs.items():
    for token_id, log_prob in zip(output.generated_tokens, output.logprobs):
        token = tokenizer.decode([token_id])
        print(f"{token} | logprob: {log_prob}")
```

### CUDA graphs

CUDA graphs eliminate CPU dispatch overhead by recording the GPU execution graph once and replaying it for batches with matching shapes. Enable them explicitly with `use_cuda_graph=True`.

```py
cb_config = ContinuousBatchingConfig(use_cuda_graph=True)
```

When active, the manager pads query and KV lengths to fixed intervals so shapes repeat and graphs reuse. Smaller values of `q_padding_interval_size` and `kv_padding_interval_size` reduce wasted compute on padding, but this means there are more unique shapes the graph has to record and store which costs more memory.

```py
cb_config = ContinuousBatchingConfig(
    use_cuda_graph=True,
    q_padding_interval_size=64,
    kv_padding_interval_size=16384,
)
```

### Async batching

Async batching overlaps CPU scheduling of the next batch with GPU computation of the current one. It requires CUDA graphs and roughly doubles the VRAM used for input tensors.

```py
cb_config = ContinuousBatchingConfig(
    use_cuda_graph=True,
    use_async_batching=True,
)
```

### Compilation

`default_compile_level` applies `torch.compile` to the model's forward passes. Compilation trades a one-time warmup cost for faster forward passes during generation. Higher levels run more aggressive optimization, which improves throughput but lengthens warmup. Keep the level low for tests and benchmark iteration, and raise it for long-running serving workloads where the warmup cost is paid back over many requests.

The level ranges from `0` to `3`. Level `0` is the default and skips compilation entirely.

| Level | `mode` | `dynamic` | Trade-off |
|---|---|---|---|
| 0 | — | — | No compilation (default), fastest startup |
| 1 | `default` | `True` | Modest speedup, short warmup |
| 2 | `max-autotune-no-cudagraphs` | `True` | More speedup, longer warmup |
| 3 | `max-autotune-no-cudagraphs` | `False` | Best throughput, longest warmup |

```py
cb_config = ContinuousBatchingConfig(default_compile_level=1)
```

The level supplies a default [CompileConfig](/docs/transformers/v5.21.0/en/internal/generation_utils#transformers.CompileConfig) to the varlen and decode execution paths. It only applies to a path that has no explicit config, so `varlen_compile_config` and `decode_compile_config` take precedence when set. Under FlashAttention, the varlen path skips compilation because `max_length_k` triggers frequent recompilation, so the level affects only the decode path in that case.

### Decode fast path

When a batch contains only decode requests (one query token per sequence), the manager can dispatch to the `flash_attn_with_kvcache` kernel instead of the variable-length kernel. This is faster than the varlen path because the kernel reads and writes the paged KV cache in-place through a block table rather than going through a manual update. See [Paged attention](./paged_attention) for kernel-level details.

The fast path is sized by `max_blocks_per_request`, which dimensions the per-request block table. By default this is auto-inferred. If `max_prompt_length` and `max_generated_length` are set on the manager, the block table is sized to fit the maximum sequence length. Otherwise, a fallback default (32 blocks per request) is used.

Set `max_blocks_per_request` to a specific value to size the block table explicitly. This is useful when you know the maximum sequence length per request and want to bound the block table memory cost.

```py
cb_config = ContinuousBatchingConfig(max_blocks_per_request=64)
```

Set `max_blocks_per_request=0` to disable the fast path and force every batch through the varlen kernel. This recovers the pre-default behavior and is useful when the fast path is unavailable for your attention implementation (the manager also disables it automatically when the underlying kernel can't be used).

```py
cb_config = ContinuousBatchingConfig(max_blocks_per_request=0)
```

The fast path relies on the `flash_attn_with_kvcache` kernel, which is available for two device and attention implementation combinations.

| Device | `attn_implementation` |
|---|---|
| CUDA | `flash_attention_2` or `flash_attention_3` |
| XPU | [flash_attention_2](https://huggingface.co/kernels-community/flash-attn2) |

For any other combination, or when the kernel can't be imported, the manager falls back to the varlen path. It logs a warning only when you set `max_blocks_per_request` explicitly.

Full attention supports kernel block tables, but sliding attention does not. If the model has any sliding attention layers, including hybrid models that also use full attention, initialization forces `max_blocks_per_request` to `0`, and a nonzero value you set is overridden. The manager then logs that the decode fast path is not available because some of the model's attention types do not support kernel block tables.

### CPU offloading

CPU offloading copies evicted KV cache blocks to a pre-allocated pinned CPU buffer when the GPU KV cache is full. After cache space becomes available, the manager copies the blocks back to the GPU and resumes the request without recomputing its prompt and generated tokens.

Set `cpu_offload_space` to the CPU swap space in GiB. The default value, `0.0`, disables CPU offloading.

```py
cb_config = ContinuousBatchingConfig(cpu_offload_space=8.0)
```

By default, `cpu_offload_space_safety_threshold=0.8` limits the requested space to 80% of available system RAM when `psutil` is installed. Set `cpu_offload_space=None` to size the swap pool from the safety threshold.

### Tensor parallel timeout

Under tensor parallelism, the manager creates a CPU communication group to coordinate request submissions, cancellations, and shutdown across ranks. `cpu_group_timeout` limits how long a collective on this group can block before the process crashes. If one rank stalls, the timeout prevents the others from waiting forever.

Set a longer timeout for workloads that issue infrequent collectives, or pass `None` to disable it.

```py
cb_config = ContinuousBatchingConfig(cpu_group_timeout=600.0)
```

### Prefix caching

When multiple requests share a common prefix, like a system prompt, the manager reuses their KV cache blocks instead of recomputing them. `allow_block_sharing` defaults to `True`. Prefix sharing is active only when every attention type allocator supports sharing, so only full attention-only models use it. Pure sliding and hybrid models turn it off even when `allow_block_sharing` is `True`.

```py
cb_config = ContinuousBatchingConfig(
    allow_block_sharing=True,  # default
)
```

## Choose an attention backend

Continuous batching supports FlashAttention, SDPA, and `"paged|eager"`. Set `attn_implementation` when you load the model.

| Backend | `attn_implementation` | Requirements |
|---|---|---|
| FlashAttention 2 | "flash_attention_2" | `flash-attn` or `kernels` package |
| FlashAttention 3 | "flash_attention_3" | `flash-attn-3` or `kernels` package |
| SDPA (PyTorch native) | "sdpa" | None |
| Eager | "paged|eager" | None |

```py
model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen3-4B",
    attn_implementation="flash_attention_2",
    device_map="auto",
    dtype=torch.bfloat16,
)
```

Continuous batching runs faster with FlashAttention than with eager or SDPA, mostly because FlashAttention doesn't need an attention mask. If the model supports FlashAttention, a model loaded with `"eager"` or `"sdpa"` switches to FlashAttention 3, or to FlashAttention 2 when FlashAttention 3 isn't installed, and logs a warning. If neither version is installed, the backend doesn't change.

To keep eager or SDPA, for example when testing, set `auto_switch_to_flash=False` in [ContinuousBatchingConfig](/docs/transformers/v5.21.0/en/main_classes/continuous_batching#transformers.ContinuousBatchingConfig). Whenever the backend stays `"eager"`, continuous batching runs `"paged|eager"` instead.

## Tensor parallelism

For models too large to fit on a single GPU, shard the weights across devices with tensor parallelism. Set the number of devices with `DistributedConfig(tp_size=N)`. Continuous batching reads the tensor parallel size from the model to size the paged KV cache per shard. See [Tensor parallelism](./tensor_parallelism) for the list of supported architectures and how sharding works.

```py
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, DistributedConfig
from transformers.generation import ContinuousBatchingConfig, GenerationConfig

distributed_config = DistributedConfig(tp_size=4)
model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen3-32B",
    attn_implementation="flash_attention_2",
    distributed_config=distributed_config,
)
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3-32B")

inputs = [tokenizer.encode(p) for p in ["What's up?", "Name a cat breed."]]
generation_config = GenerationConfig(max_new_tokens=64, eos_token_id=tokenizer.eos_token_id)

outputs = model.generate_batch(inputs=inputs, generation_config=generation_config)
```

Launch the script with `torchrun`, setting `--nproc-per-node` to the number of GPUs you want to shard across.

```shell
torchrun --nproc-per-node 4 cb_tp.py
```

The tensor parallel size must divide the model's `num_key_value_heads` (check the model config). The paged cache raises an error at startup otherwise, so choose an appropriate `--nproc-per-node`.

> [!WARNING]
> Don't set `device_map` with `distributed_config`. The two conflict because `device_map` places whole modules on specific GPUs, while tensor parallelism shards those same parameters across all GPUs.

## Sliding window attention

Models with sliding window attention (Mistral, Gemma 2) work with continuous batching. Sliding layers use `SlidingAttentionCacheAllocator`, which keeps only a fixed window of recent tokens by recycling a small set of pages. Attention type comes from the model's `layer_types` or `sliding_window`, not from [ContinuousBatchingConfig](/docs/transformers/v5.21.0/en/main_classes/continuous_batching#transformers.ContinuousBatchingConfig).

```py
from transformers import AutoConfig, AutoModelForCausalLM

config = AutoConfig.from_pretrained("google/gemma-2-2b")
config.sliding_window = 4096

model = AutoModelForCausalLM.from_pretrained(
    "google/gemma-2-2b",
    config=config,
    attn_implementation="sdpa",
    device_map="auto",
    dtype=torch.bfloat16,
)
```

[Prefix caching](#prefix-caching) and the [decode fast path](#decode-fast-path) are disabled whenever sliding attention is present, including hybrid models that also have full attention layers.

## Next steps

- The [Continuous batching blog post](https://huggingface.co/blog/continuous_batching) covers KV caching, chunked prefill, and dynamic scheduling with performance benchmark numbers.
- For a deeper look at how the continuous batching system works, see the [Continuous batching architecture](./continuous_batching_architecture) doc.