CPU

CPUs run Transformers models with the device string cpu, without an accelerator. They work for inference and training when GPUs aren’t available, or when you want a cost-effective setup.

Run inference

Models load on CPU by default when you don’t pass device_map, and pipeline() uses the CPU when no accelerator is available. Pass device_map="cpu" or device="cpu" to use the CPU on a machine that also has an accelerator.

from transformers import pipeline

generator = pipeline("text-generation", model="Qwen/Qwen3-0.6B", device="cpu")
print(generator("CPUs are", max_new_tokens=50, do_sample=False)[0]["generated_text"])

Train with Trainer

CPUs with bf16 support, such as recent Intel and AMD processors, can use bf16 mixed precision training with PyTorch’s AMP, which reduces memory usage and speeds up training.

The examples below cover three setups, from a single CPU to several machines when one CPU is too slow:

• a single CPU
• multiple processes on one machine (one per CPU socket)
• multiple processes across several machines

The single-node and multi-node examples launch processes with mpirun from Intel MPI, part of the Intel oneAPI HPC Toolkit, and Trainer runs DDP over the gloo backend.

from transformers import TrainingArguments

training_args = TrainingArguments(
    output_dir="./outputs",
    bf16=True,
    use_cpu=True,
)

Tune CPU performance

Preload a faster memory allocator and Intel’s OpenMP runtime to reduce overhead in CPU training. Install google-perftools (for libtcmalloc) and the intel-openmp Python package (for libiomp5), then add both libraries to the LD_PRELOAD environment variable before you launch training.

Set OMP_NUM_THREADS to the number of cores each process can use. In a container or a cluster job with CPU limits, that’s the number of allocated CPU units, so leave some cores unallocated for system processes.

Next steps

• Read the Accelerating PyTorch Transformers with Intel Sapphire Rapids blog post for a deeper look at BF16 performance on modern Intel hardware.
• Read DDP for more on the distributed strategy the multi-process examples use.
• Read the PyTorch CPU threading notes to learn how OMP_NUM_THREADS affects performance.
• Read the Kubeflow PyTorchJob guide to run distributed CPU training on Kubernetes.