Resource Management for Pods and ContainersWhen you specify a Pod, you can optionally specify how much of each resource a container needs. The most common resources to specify are CPU and memory (RAM); there are others.When you specify the resource request for containers in a Pod, the kube-scheduler uses this information to decide which node to place the Pod on. When you specify a resource limit for a container, the kubelet enforces those limits so that the running container is not allowed to use more of that resource than the limit you set. The kubelet also reserves at least the request amount of that system resource specifically for that container to use.Requests and limitsIf the node where a Pod is running has enough of a resource available, it's possible (and allowed) for a container to use more resource than its request for that resource specifies.For example, if you set a memory request of 256 MiB for a container, and that container is in a Pod scheduled to a Node with 8GiB of memory and no other Pods, then the container can try to use more RAM.Limits are a different story. Both cpu and memory limits are applied by the kubelet (and container runtime), and are ultimately enforced by the kernel. On Linux nodes, the Linux kernel enforces limits with cgroups. The behavior of cpu and memory limit enforcement is slightly different.cpu limits are enforced by CPU throttling. When a container approaches its cpu limit, the kernel will restrict access to the CPU corresponding to the container's limit. Thus, a cpu limit is a hard limit the kernel enforces. Containers may not use more CPU than is specified in their cpu limit.memory limits are enforced by the kernel with out of memory (OOM) kills. When a container uses more than its memory limit, the kernel may terminate it. However, terminations only happen when the kernel detects memory pressure. Thus, a container that over allocates memory may not be immediately killed. This means memory limits are enforced reactively. A container may use more memory than its memory limit, but if it does, it may get killed.Note:There is an alpha feature MemoryQoS which attempts to add more preemptive limit enforcement for memory (as opposed to reactive enforcement by the OOM killer). However, this effort is stalled due to a potential livelock situation a memory hungry can cause.Note:If you specify a limit for a resource, but do not specify any request, and no admission-time mechanism has applied a default request for that resource, then Kubernetes copies the limit you specified and uses it as the requested value for the resource.Resource typesCPU and memory are each a resource type. A resource type has a base unit. CPU represents compute processing and is specified in units of Kubernetes CPUs. Memory is specified in units of bytes. For Linux workloads, you can specify huge page resources. Huge pages are a Linux-specific feature where the node kernel allocates blocks of memory that are much larger than the default page size.For example, on a system where the default page size is 4KiB, you could specify a limit, hugepages-2Mi: 80Mi. If the container tries allocating over 40 2MiB huge pages (a total of 80 MiB), that allocation fails.Note:You cannot overcommit hugepages-* resources. This is different from the memory and cpu resources.CPU and memory are collectively referred to as compute resources, or resources. Compute resources are measurable quantities that can be requested, allocated, and consumed. They are distinct from API resources. API resources, such as Pods and Services are objects that can be read and modified through the Kubernetes API server.Resource requests and limits of Pod and containerFor each container, you can specify resource limits and requests, including the following:spec.containers[].resources.limits.cpuspec.containers[].resources.limits.memoryspec.containers[].resources.limits.hugepages-<size>spec.containers[].resources.requests.cpuspec.containers[].resources.requests.memoryspec.containers[].resources.requests.hugepages-<size>Although you can only specify requests and limits for individual containers, it is also useful to think about the overall resource requests and limits for a Pod. For a particular resource, a Pod resource request/limit is the sum of the resource requests/limits of that type for each container in the Pod.Pod-level resource specificationFEATURE STATE: Kubernetes v1.34 [beta] (enabled by default: true)Provided your cluster has the PodLevelResources feature gate enabled, you can specify resource requests and limits at the Pod level. At the Pod level, Kubernetes 1.34 only supports resource requests or limits for specific resource types: cpu and / or memory and / or hugepages. With this feature, Kubernetes allows you to declare an overall resource budget for the Pod, which is especially helpful when dealing with a large number of containers where it can be difficult to accurately gauge individual resource needs. Additionally, it enables containers within a Pod to share idle resources with each other, improving resource utilization.For a Pod, you can specify resource limits and requests for CPU and memory by including the following:spec.resources.limits.cpuspec.resources.limits.memoryspec.resources.limits.hugepages-<size>spec.resources.requests.cpuspec.resources.requests.memoryspec.resources.requests.hugepages-<size>Resource units in KubernetesCPU resource unitsLimits and requests for CPU resources are measured in cpu units. In Kubernetes, 1 CPU unit is equivalent to 1 physical CPU core, or 1 virtual core, depending on whether the node is a physical host or a virtual machine running inside a physical machine.Fractional requests are allowed. When you define a container with spec.containers[].resources.requests.cpu set to 0.5, you are requesting half as much CPU time compared to if you asked for 1.0 CPU. For CPU resource units, the quantity expression 0.1 is equivalent to the expression 100m, which can be read as "one hundred millicpu". Some people say "one hundred millicores", and this is understood to mean the same thing.CPU resource is always specified as an absolute amount of resource, never as a relative amount. For example, 500m CPU represents the roughly same amount of computing power whether that container runs on a single-core, dual-core, or 48-core machine.Note:Kubernetes doesn't allow you to specify CPU resources with a precision finer than 1m or 0.001 CPU. To avoid accidentally using an invalid CPU quantity, it's useful to specify CPU units using the milliCPU form instead of the decimal form when using less than 1 CPU unit.For example, you have a Pod that uses 5m or 0.005 CPU and would like to decrease its CPU resources. By using the decimal form, it's harder to spot that 0.0005 CPU is an invalid value, while by using the milliCPU form, it's easier to spot that 0.5m is an invalid value.Memory resource unitsLimits and requests for memory are measured in bytes. You can express memory as a plain integer or as a fixed-point number using one of these quantity suffixes: E, P, T, G, M, k. You can also use the power-of-two equivalents: Ei, Pi, Ti, Gi, Mi, Ki. For example, the following represent roughly the same value:128974848, 129e6, 129M, 128974848000m, 123Mi Pay attention to the case of the suffixes. If you request 400m of memory, this is a request for 0.4 bytes. Someone who types that probably meant to ask for 400 mebibytes (400Mi) or 400 megabytes (400M).Container resources exampleThe following Pod has two containers. Both containers are defined with a request for 0.25 CPU and 64MiB (226 bytes) of memory. Each container has a limit of 0.5 CPU and 128MiB of memory. You can say the Pod has a request of 0.5 CPU and 128 MiB of memory, and a limit of 1 CPU and 256MiB of memory.--- apiVersion: v1 kind: Pod metadata: name: frontend spec: containers: - name: app image: images.my-company.example/app:v4 resources: requests: memory: "64Mi" cpu: "250m" limits: memory: "128Mi" cpu: "500m" - name: log-aggregator image: images.my-company.example/log-aggregator:v6 resources: requests: memory: "64Mi" cpu: "250m" limits: memory: "128Mi" cpu: "500m" Pod resources exampleFEATURE STATE: Kubernetes v1.34 [beta] (enabled by default: true)This feature can be enabled by setting the PodLevelResources feature gate. The following Pod has an explicit request of 1 CPU and 100 MiB of memory, and an explicit limit of 1 CPU and 200 MiB of memory. The pod-resources-demo-ctr-1 container has explicit requests and limits set. However, the pod-resources-demo-ctr-2 container will simply share the resources available within the Pod resource boundaries, as it does not have explicit requests and limits set.pods/resource/pod-level-resources.yaml apiVersion: v1 kind: Pod metadata: name: pod-resources-demo namespace: pod-resources-example spec: resources: limits: cpu: "1" memory: "200Mi" requests: cpu: "1" memory: "100Mi" containers: - name: pod-resources-demo-ctr-1 image: nginx resources: limits: cpu: "0.5" memory: "100Mi" requests: cpu: "0.5" memory: "50Mi" - name: pod-resources-demo-ctr-2 image: fedora command: - sleep - inf How Pods with resource requests are scheduledWhen you create a Pod, the Kubernetes scheduler selects a node for the Pod to run on. Each node has a maximum capacity for each of the resource types: the amount of CPU and memory it can provide for Pods. The scheduler ensures that, for each resource type, the sum of the resource requests of the scheduled containers is less than the capacity of the node. Note that although actual memory or CPU resource usage on nodes is very low, the scheduler still refuses to place a Pod on a node if the capacity check fails. This protects against a resource shortage on a node when resource usage later increases, for example, during a daily peak in request rate.How Kubernetes applies resource requests and limitsWhen the kubelet starts a container as part of a Pod, the kubelet passes that container's requests and limits for memory and CPU to the container runtime.On Linux, the container runtime typically configures kernel cgroups that apply and enforce the limits you defined.The CPU limit defines a hard ceiling on how much CPU time the container can use. During each scheduling interval (time slice), the Linux kernel checks to see if this limit is exceeded; if so, the kernel waits before allowing that cgroup to resume execution.The CPU request typically defines a weighting. If several different containers (cgroups) want to run on a contended system, workloads with larger CPU requests are allocated more CPU time than workloads with small requests.The memory request is mainly used during (Kubernetes) Pod scheduling. On a node that uses cgroups v2, the container runtime might use the memory request as a hint to set memory.min and memory.low.The memory limit defines a memory limit for that cgroup. If the container tries to allocate more memory than this limit, the Linux kernel out-of-memory subsystem activates and, typically, intervenes by stopping one of the processes in the container that tried to allocate memory. If that process is the container's PID 1, and the container is marked as restartable, Kubernetes restarts the container.The memory limit for the Pod or container can also apply to pages in memory backed volumes, such as an emptyDir. The kubelet tracks tmpfs emptyDir volumes as container memory use, rather than as local ephemeral storage. When using memory backed emptyDir, be sure to check the notes below.If a container exceeds its memory request and the node that it runs on becomes short of memory overall, it is likely that the Pod the container belongs to will be evicted.A container might or might not be allowed to exceed its CPU limit for extended periods of time. However, container runtimes don't terminate Pods or containers for excessive CPU usage.To determine whether a container cannot be scheduled or is being killed due to resource limits, see the Troubleshooting section.Monitoring compute & memory resource usageThe kubelet reports the resource usage of a Pod as part of the Pod status.If optional tools for monitoring are available in your cluster, then Pod resource usage can be retrieved either from the Metrics API directly or from your monitoring tools.Considerations for memory backed emptyDir volumesCaution:If you do not specify a sizeLimit for an emptyDir volume, that volume may consume up to that pod's memory limit (Pod.spec.containers[].resources.limits.memory). If you do not set a memory limit, the pod has no upper bound on memory consumption, and can consume all available memory on the node. Kubernetes schedules pods based on resource requests (Pod.spec.containers[].resources.requests) and will not consider memory usage above the request when deciding if another pod can fit on a given node. This can result in a denial of service and cause the OS to do out-of-memory (OOM) handling. It is possible to create any number of emptyDirs that could potentially consume all available memory on the node, making OOM more likely.From the perspective of memory management, there are some similarities between when a process uses memory as a work area and when using memory-backed emptyDir. But when using memory as a volume, like memory-backed emptyDir, there are additional points below that you should be careful of:Files stored on a memory-backed volume are almost entirely managed by the user application. Unlike when used as a work area for a process, you can not rely on things like language-level garbage collection.The purpose of writing files to a volume is to save data or pass it between applications. Neither Kubernetes nor the OS may automatically delete files from a volume, so memory used by those files can not be reclaimed when the system or the pod are under memory pressure.A memory-backed emptyDir is useful because of its performance, but memory is generally much smaller in size and much higher in cost than other storage media, such as disks or SSDs. Using large amounts of memory for emptyDir volumes may affect the normal operation of your pod or of the whole node, so should be used carefully.If you are administering a cluster or namespace, you can also set ResourceQuota that limits memory use; you may also want to define a LimitRange for additional enforcement. If you specify a spec.containers[].resources.limits.memory for each Pod, then the maximum size of an emptyDir volume will be the pod's memory limit.As an alternative, a cluster administrator can enforce size limits for emptyDir volumes in new Pods using a policy mechanism such as ValidationAdmissionPolicy.Local ephemeral storageFEATURE STATE: Kubernetes v1.25 [stable]Nodes have local ephemeral storage, backed by locally-attached writeable devices or, sometimes, by RAM. "Ephemeral" means that there is no long-term guarantee about durability.Pods use ephemeral local storage for scratch space, caching, and for logs. The kubelet can provide scratch space to Pods using local ephemeral storage to mount emptyDir volumes into containers.The kubelet also uses this kind of storage to hold node-level container logs, container images, and the writable layers of running containers.Caution:If a node fails, the data in its ephemeral storage can be lost. Your applications cannot expect any performance SLAs (disk IOPS for example) from local ephemeral storage.Note:To make the resource quota work on ephemeral-storage, two things need to be done:An admin sets the resource quota for ephemeral-storage in a namespace.A user needs to specify limits for the ephemeral-storage resource in the Pod spec.If the user doesn't specify the ephemeral-storage resource limit in the Pod spec, the resource quota is not enforced on ephemeral-storage.Kubernetes lets you track, reserve and limit the amount of ephemeral local storage a Pod can consume.Configurations for local ephemeral storageKubernetes supports two ways to configure local ephemeral storage on a node:In this configuration, you place all different kinds of ephemeral local data (emptyDir volumes, writeable layers, container images, logs) into one filesystem. The most effective way to configure the kubelet means dedicating this filesystem to Kubernetes (kubelet) data.The kubelet also writes node-level container logs and treats these similarly to ephemeral local storage.The kubelet writes logs to files inside its configured log directory (/var/log by default); and has a base directory for other locally stored data (/var/lib/kubelet by default).Typically, both /var/lib/kubelet and /var/log are on the system root filesystem, and the kubelet is designed with that layout in mind.Your node can have as many other filesystems, not used for Kubernetes, as you like.You have a filesystem on the node that you're using for ephemeral data that comes from running Pods: logs, and emptyDir volumes. You can use this filesystem for other data (for example: system logs not related to Kubernetes); it can even be the root filesystem.The kubelet also writes node-level container logs into the first filesystem, and treats these similarly to ephemeral local storage.You also use a separate filesystem, backed by a different logical storage device. In this configuration, the directory where you tell the kubelet to place container image layers and writeable layers is on this second filesystem.The first filesystem does not hold any image layers or writeable layers.Your node can have as many other filesystems, not used for Kubernetes, as you like.The kubelet can measure how much local storage it is using. It does this provided that you have set up the node using one of the supported configurations for local ephemeral storage.If you have a different configuration, then the kubelet does not apply resource limits for ephemeral local storage.Note:The kubelet tracks tmpfs emptyDir volumes as container memory use, rather than as local ephemeral storage.Note:The kubelet will only track the root filesystem for ephemeral storage. OS layouts that mount a separate disk to /var/lib/kubelet or /var/lib/containers will not report ephemeral storage correctly.Setting requests and limits for local ephemeral storageYou can specify ephemeral-storage for managing local ephemeral storage. Each container of a Pod can specify either or both of the following:spec.containers[].resources.limits.ephemeral-storagespec.containers[].resources.requests.ephemeral-storageLimits and requests for ephemeral-storage are measured in byte quantities. You can express storage as a plain integer or as a fixed-point number using one of these suffixes: E, P, T, G, M, k. You can also use the power-of-two equivalents: Ei, Pi, Ti, Gi, Mi, Ki. For example, the following quantities all represent roughly the same value:128974848129e6129M123MiPay attention to the case of the suffixes. If you request 400m of ephemeral-storage, this is a request for 0.4 bytes. Someone who types that probably meant to ask for 400 mebibytes (400Mi) or 400 megabytes (400M).In the following example, the Pod has two containers. Each container has a request of 2GiB of local ephemeral storage. Each container has a limit of 4GiB of local ephemeral storage. Therefore, the Pod has a request of 4GiB of local ephemeral storage, and a limit of 8GiB of local ephemeral storage. 500Mi of that limit could be consumed by the emptyDir volume.apiVersion: v1 kind: Pod metadata: name: frontend spec: containers: - name: app image: images.my-company.example/app:v4 resources: requests: ephemeral-storage: "2Gi" limits: ephemeral-storage: "4Gi" volumeMounts: - name: ephemeral mountPath: "/tmp" - name: log-aggregator image: images.my-company.example/log-aggregator:v6 resources: requests: ephemeral-storage: "2Gi" limits: ephemeral-storage: "4Gi" volumeMounts: - name: ephemeral mountPath: "/tmp" volumes: - name: ephemeral emptyDir: sizeLimit: 500Mi How Pods with ephemeral-storage requests are scheduledWhen you create a Pod, the Kubernetes scheduler selects a node for the Pod to run on. Each node has a maximum amount of local ephemeral storage it can provide for Pods. For more information, see Node Allocatable.The scheduler ensures that the sum of the resource requests of the scheduled containers is less than the capacity of the node.Ephemeral storage consumption managementIf the kubelet is managing local ephemeral storage as a resource, then the kubelet measures storage use in:emptyDir volumes, except tmpfs emptyDir volumesdirectories holding node-level logswriteable container layersIf a Pod is using more ephemeral storage than you allow it to, the kubelet sets an eviction signal that triggers Pod eviction.For container-level isolation, if a container's writable layer and log usage exceeds its storage limit, the kubelet marks the Pod for eviction.For pod-level isolation the kubelet works out an overall Pod storage limit by summing the limits for the containers in that Pod. In this case, if the sum of the local ephemeral storage usage from all containers and also the Pod's emptyDir volumes exceeds the overall Pod storage limit, then the kubelet also marks the Pod for eviction.Caution:If the kubelet is not measuring local ephemeral storage, then a Pod that exceeds its local storage limit will not be evicted for breaching local storage resource limits.However, if the filesystem space for writeable container layers, node-level logs, or emptyDir volumes falls low, the node taints itself as short on local storage and this taint triggers eviction for any Pods that don't specifically tolerate the taint.See the supported configurations for ephemeral local storage.The kubelet supports different ways to measure Pod storage use:The kubelet performs regular, scheduled checks that scan each emptyDir volume, container log directory, and writeable container layer.The scan measures how much space is used.Note:In this mode, the kubelet does not track open file descriptors for deleted files.If you (or a container) create a file inside an emptyDir volume, something then opens that file, and you delete the file while it is still open, then the inode for the deleted file stays until you close that file but the kubelet does not categorize the space as in use.FEATURE STATE: Kubernetes v1.31 [beta] (enabled by default: false)Project quotas are an operating-system level feature for managing storage use on filesystems. With Kubernetes, you can enable project quotas for monitoring storage use. Make sure that the filesystem backing the emptyDir volumes, on the node, provides project quota support. For example, XFS and ext4fs offer project quotas.Note:Project quotas let you monitor storage use; they do not enforce limits.Kubernetes uses project IDs starting from 1048576. The IDs in use are registered in /etc/projects and /etc/projid. If project IDs in this range are used for other purposes on the system, those project IDs must be registered in /etc/projects and /etc/projid so that Kubernetes does not use them.Quotas are faster and more accurate than directory scanning. When a directory is assigned to a project, all files created under a directory are created in that project, and the kernel merely has to keep track of how many blocks are in use by files in that project. If a file is created and deleted, but has an open file descriptor, it continues to consume space. Quota tracking records that space accurately whereas directory scans overlook the storage used by deleted files.To use quotas to track a pod's resource usage, the pod must be in a user namespace. Within user namespaces, the kernel restricts changes to projectIDs on the filesystem, ensuring the reliability of storage metrics calculated by quotas.If you want to use project quotas, you should:Enable the LocalStorageCapacityIsolationFSQuotaMonitoring=true feature gate using the featureGates field in the kubelet configuration.Ensure the UserNamespacesSupport feature gate is enabled, and that the kernel, CRI implementation and OCI runtime support user namespaces.Ensure that the root filesystem (or optional runtime filesystem) has project quotas enabled. All XFS filesystems support project quotas. For ext4 filesystems, you need to enable the project quota tracking feature while the filesystem is not mounted.# For ext4, with /dev/block-device not mounted sudo tune2fs -O project -Q prjquota /dev/block-device Ensure that the root filesystem (or optional runtime filesystem) is mounted with project quotas enabled. For both XFS and ext4fs, the mount option is named prjquota.If you don't want to use project quotas, you should:Disable the LocalStorageCapacityIsolationFSQuotaMonitoring feature gate using the featureGates field in the kubelet configuration.Extended resourcesExtended resources are fully-qualified resource names outside the kubernetes.io domain. They allow cluster operators to advertise and users to consume the non-Kubernetes-built-in resources.There are two steps required to use Extended Resources. First, the cluster operator must advertise an Extended Resource. Second, users must request the Extended Resource in Pods.Managing extended resourcesNode-level extended resourcesNode-level extended resources are tied to nodes.Device plugin managed resourcesSee Device Plugin for how to advertise device plugin managed resources on each node.Other resourcesTo advertise a new node-level extended resource, the cluster operator can submit a PATCH HTTP request to the API server to specify the available quantity in the status.capacity for a node in the cluster. After this operation, the node's status.capacity will include a new resource. The status.allocatable field is updated automatically with the new resource asynchronously by the kubelet.Because the scheduler uses the node's status.allocatable value when evaluating Pod fitness, the scheduler only takes account of the new value after that asynchronous update. There may be a short delay between patching the node capacity with a new resource and the time when the first Pod that requests the resource can be scheduled on that node.Example:Here is an example showing how to use curl to form an HTTP request that advertises five "example.com/foo" resources on node k8s-node-1 whose master is k8s-master.curl --header "Content-Type: application/json-patch+json" \ --request PATCH \ --data '[{"op": "add", "path": "/status/capacity/example.com~1foo", "value": "5"}]' \ http://k8s-master:8080/api/v1/nodes/k8s-node-1/status Note:In the preceding request, ~1 is the encoding for the character / in the patch path. The operation path value in JSON-Patch is interpreted as a JSON-Pointer. For more details, see IETF RFC 6901, section 3.Cluster-level extended resourcesCluster-level extended resources are not tied to nodes. They are usually managed by scheduler extenders, which handle the resource consumption and resource quota.You can specify the extended resources that are handled by scheduler extenders in scheduler configurationExample:The following configuration for a scheduler policy indicates that the cluster-level extended resource "example.com/foo" is handled by the scheduler extender.The scheduler sends a Pod to the scheduler extender only if the Pod requests "example.com/foo".The ignoredByScheduler field specifies that the scheduler does not check the "example.com/foo" resource in its PodFitsResources predicate.{ "kind": "Policy", "apiVersion": "v1", "extenders": [ { "urlPrefix":"<extender-endpoint>", "bindVerb": "bind", "managedResources": [ { "name": "example.com/foo", "ignoredByScheduler": true } ] } ] } Extended resources allocation by DRAExtended resources allocation by DRA allows cluster administrators to specify an extendedResourceName in DeviceClass, then the devices matching the DeviceClass can be requested from a pod's extended resource requests. Read more about Extended Resource allocation by DRA.Consuming extended resourcesUsers can consume extended resources in Pod specs like CPU and memory. The scheduler takes care of the resource accounting so that no more than the available amount is simultaneously allocated to Pods.The API server restricts quantities of extended resources to whole numbers. Examples of valid quantities are 3, 3000m and 3Ki. Examples of invalid quantities are 0.5 and 1500m (because 1500m would result in 1.5).Note:Extended resources replace Opaque Integer Resources. Users can use any domain name prefix other than kubernetes.io which is reserved.To consume an extended resource in a Pod, include the resource name as a key in the spec.containers[].resources.limits map in the container spec.Note:Extended resources cannot be overcommitted, so request and limit must be equal if both are present in a container spec.A Pod is scheduled only if all of the resource requests are satisfied, including CPU, memory and any extended resources. The Pod remains in the PENDING state as long as the resource request cannot be satisfied.Example:The Pod below requests 2 CPUs and 1 "example.com/foo" (an extended resource).apiVersion: v1 kind: Pod metadata: name: my-pod spec: containers: - name: my-container image: myimage resources: requests: cpu: 2 example.com/foo: 1 limits: example.com/foo: 1 PID limitingProcess ID (PID) limits allow for the configuration of a kubelet to limit the number of PIDs that a given Pod can consume. See PID Limiting for information.TroubleshootingMy Pods are pending with event message FailedSchedulingIf the scheduler cannot find any node where a Pod can fit, the Pod remains unscheduled until a place can be found. An Event is produced each time the scheduler fails to find a place for the Pod. You can use kubectl to view the events for a Pod; for example:kubectl describe pod frontend | grep -A 9999999999 Events Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedScheduling 23s default-scheduler 0/42 nodes available: insufficient cpu In the preceding example, the Pod named "frontend" fails to be scheduled due to insufficient CPU resource on any node. Similar error messages can also suggest failure due to insufficient memory (PodExceedsFreeMemory). In general, if a Pod is pending with a message of this type, there are several things to try:Add more nodes to the cluster.Terminate unneeded Pods to make room for pending Pods.Check that the Pod is not larger than all the nodes. For example, if all the nodes have a capacity of cpu: 1, then a Pod with a request of cpu: 1.1 will never be scheduled.Check for node taints. If most of your nodes are tainted, and the new Pod does not tolerate that taint, the scheduler only considers placements onto the remaining nodes that don't have that taint.You can check node capacities and amounts allocated with the kubectl describe nodes command. For example:kubectl describe nodes e2e-test-node-pool-4lw4 Name: e2e-test-node-pool-4lw4 [ ... lines removed for clarity ...] Capacity: cpu: 2 memory: 7679792Ki pods: 110 Allocatable: cpu: 1800m memory: 7474992Ki pods: 110 [ ... lines removed for clarity ...] Non-terminated Pods: (5 in total) Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits --------- ---- ------------ ---------- --------------- ------------- kube-system fluentd-gcp-v1.38-28bv1 100m (5%) 0 (0%) 200Mi (2%) 200Mi (2%) kube-system kube-dns-3297075139-61lj3 260m (13%) 0 (0%) 100Mi (1%) 170Mi (2%) kube-system kube-proxy-e2e-test-... 100m (5%) 0 (0%) 0 (0%) 0 (0%) kube-system monitoring-influxdb-grafana-v4-z1m12 200m (10%) 200m (10%) 600Mi (8%) 600Mi (8%) kube-system node-problem-detector-v0.1-fj7m3 20m (1%) 200m (10%) 20Mi (0%) 100Mi (1%) Allocated resources: (Total limits may be over 100 percent, i.e., overcommitted.) CPU Requests CPU Limits Memory Requests Memory Limits ------------ ---------- --------------- ------------- 680m (34%) 400m (20%) 920Mi (11%) 1070Mi (13%) In the preceding output, you can see that if a Pod requests more than 1.120 CPUs or more than 6.23Gi of memory, that Pod will not fit on the node.By looking at the “Pods” section, you can see which Pods are taking up space on the node.The amount of resources available to Pods is less than the node capacity because system daemons use a portion of the available resources. Within the Kubernetes API, each Node has a .status.allocatable field (see NodeStatus for details).The .status.allocatable field describes the amount of resources that are available to Pods on that node (for example: 15 virtual CPUs and 7538 MiB of memory). For more information on node allocatable resources in Kubernetes, see Reserve Compute Resources for System Daemons.You can configure resource quotas to limit the total amount of resources that a namespace can consume. Kubernetes enforces quotas for objects in particular namespace when there is a ResourceQuota in that namespace. For example, if you assign specific namespaces to different teams, you can add ResourceQuotas into those namespaces. Setting resource quotas helps to prevent one team from using so much of any resource that this over-use affects other teams.You should also consider what access you grant to that namespace: full write access to a namespace allows someone with that access to remove any resource, including a configured ResourceQuota.My container is terminatedYour container might get terminated because it is resource-starved. To check whether a container is being killed because it is hitting a resource limit, call kubectl describe pod on the Pod of interest:kubectl describe pod simmemleak-hra99 The output is similar to:Name: simmemleak-hra99 Namespace: default Image(s): saadali/simmemleak Node: kubernetes-node-tf0f/10.240.216.66 Labels: name=simmemleak Status: Running Reason: Message: IP: 10.244.2.75 Containers: simmemleak: Image: saadali/simmemleak:latest Limits: cpu: 100m memory: 50Mi State: Running Started: Tue, 07 Jul 2019 12:54:41 -0700 Last State: Terminated Reason: OOMKilled Exit Code: 137 Started: Fri, 07 Jul 2019 12:54:30 -0700 Finished: Fri, 07 Jul 2019 12:54:33 -0700 Ready: False Restart Count: 5 Conditions: Type Status Ready False Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 42s default-scheduler Successfully assigned simmemleak-hra99 to kubernetes-node-tf0f Normal Pulled 41s kubelet Container image "saadali/simmemleak:latest" already present on machine Normal Created 41s kubelet Created container simmemleak Normal Started 40s kubelet Started container simmemleak Normal Killing 32s kubelet Killing container with id ead3fb35-5cf5-44ed-9ae1-488115be66c6: Need to kill Pod In the preceding example, the Restart Count: 5 indicates that the simmemleak container in the Pod was terminated and restarted five times (so far). The OOMKilled reason shows that the container tried to use more memory than its limit.Your next step might be to check the application code for a memory leak. If you find that the application is behaving how you expect, consider setting a higher memory limit (and possibly request) for that container.What's nextGet hands-on experience assigning Memory resources to containers and Pods.Get hands-on experience assigning CPU resources to containers and Pods.Read how the API reference defines a container and its resource requirementsRead about project quotas in XFSRead more about the kube-scheduler configuration reference (v1)Read more about Quality of Service classes for PodsRead more about Extended Resource allocation by DRA


Configure Minimum and Maximum CPU Constraints for a NamespaceDefine a range of valid CPU resource limits for a namespace, so that every new Pod in that namespace falls within the range you configure.This page shows how to set minimum and maximum values for the CPU resources used by containers and Pods in a namespace. You specify minimum and maximum CPU values in a LimitRange object. If a Pod does not meet the constraints imposed by the LimitRange, it cannot be created in the namespace.Before you beginYou need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. It is recommended to run this tutorial on a cluster with at least two nodes that are not acting as control plane hosts. If you do not already have a cluster, you can create one by using minikube or you can use one of these Kubernetes playgrounds:iximiuz LabsKillercodaKodeKloudPlay with KubernetesYou must have access to create namespaces in your cluster.Each node in your cluster must have at least 1.0 CPU available for Pods. See meaning of CPU to learn what Kubernetes means by “1 CPU”.Create a namespaceCreate a namespace so that the resources you create in this exercise are isolated from the rest of your cluster.kubectl create namespace constraints-cpu-example Create a LimitRange and a PodHere's a manifest for an example LimitRange:admin/resource/cpu-constraints.yaml apiVersion: v1 kind: LimitRange metadata: name: cpu-min-max-demo-lr spec: limits: - max: cpu: "800m" min: cpu: "200m" type: Container Create the LimitRange:kubectl apply -f https://k8s.io/examples/admin/resource/cpu-constraints.yaml --namespace=constraints-cpu-example View detailed information about the LimitRange:kubectl get limitrange cpu-min-max-demo-lr --output=yaml --namespace=constraints-cpu-example The output shows the minimum and maximum CPU constraints as expected. But notice that even though you didn't specify default values in the configuration file for the LimitRange, they were created automatically.limits: - default: cpu: 800m defaultRequest: cpu: 800m max: cpu: 800m min: cpu: 200m type: Container Now whenever you create a Pod in the constraints-cpu-example namespace (or some other client of the Kubernetes API creates an equivalent Pod), Kubernetes performs these steps:If any container in that Pod does not specify its own CPU request and limit, the control plane assigns the default CPU request and limit to that container.Verify that every container in that Pod specifies a CPU request that is greater than or equal to 200 millicpu.Verify that every container in that Pod specifies a CPU limit that is less than or equal to 800 millicpu.Note:When creating a LimitRange object, you can specify limits on huge-pages or GPUs as well. However, when both default and defaultRequest are specified on these resources, the two values must be the same.Here's a manifest for a Pod that has one container. The container manifest specifies a CPU request of 500 millicpu and a CPU limit of 800 millicpu. These satisfy the minimum and maximum CPU constraints imposed by the LimitRange for this namespace.admin/resource/cpu-constraints-pod.yaml apiVersion: v1 kind: Pod metadata: name: constraints-cpu-demo spec: containers: - name: constraints-cpu-demo-ctr image: nginx resources: limits: cpu: "800m" requests: cpu: "500m" Create the Pod:kubectl apply -f https://k8s.io/examples/admin/resource/cpu-constraints-pod.yaml --namespace=constraints-cpu-example Verify that the Pod is running and that its container is healthy:kubectl get pod constraints-cpu-demo --namespace=constraints-cpu-example View detailed information about the Pod:kubectl get pod constraints-cpu-demo --output=yaml --namespace=constraints-cpu-example The output shows that the Pod's only container has a CPU request of 500 millicpu and CPU limit of 800 millicpu. These satisfy the constraints imposed by the LimitRange.resources: limits: cpu: 800m requests: cpu: 500m Delete the Podkubectl delete pod constraints-cpu-demo --namespace=constraints-cpu-example Attempt to create a Pod that exceeds the maximum CPU constraintHere's a manifest for a Pod that has one container. The container specifies a CPU request of 500 millicpu and a cpu limit of 1.5 cpu.admin/resource/cpu-constraints-pod-2.yaml apiVersion: v1 kind: Pod metadata: name: constraints-cpu-demo-2 spec: containers: - name: constraints-cpu-demo-2-ctr image: nginx resources: limits: cpu: "1.5" requests: cpu: "500m" Attempt to create the Pod:kubectl apply -f https://k8s.io/examples/admin/resource/cpu-constraints-pod-2.yaml --namespace=constraints-cpu-example The output shows that the Pod does not get created, because it defines an unacceptable container. That container is not acceptable because it specifies a CPU limit that is too large:Error from server (Forbidden): error when creating "examples/admin/resource/cpu-constraints-pod-2.yaml": pods "constraints-cpu-demo-2" is forbidden: maximum cpu usage per Container is 800m, but limit is 1500m. Attempt to create a Pod that does not meet the minimum CPU requestHere's a manifest for a Pod that has one container. The container specifies a CPU request of 100 millicpu and a CPU limit of 800 millicpu.admin/resource/cpu-constraints-pod-3.yaml apiVersion: v1 kind: Pod metadata: name: constraints-cpu-demo-3 spec: containers: - name: constraints-cpu-demo-3-ctr image: nginx resources: limits: cpu: "800m" requests: cpu: "100m" Attempt to create the Pod:kubectl apply -f https://k8s.io/examples/admin/resource/cpu-constraints-pod-3.yaml --namespace=constraints-cpu-example The output shows that the Pod does not get created, because it defines an unacceptable container. That container is not acceptable because it specifies a CPU request that is lower than the enforced minimum:Error from server (Forbidden): error when creating "examples/admin/resource/cpu-constraints-pod-3.yaml": pods "constraints-cpu-demo-3" is forbidden: minimum cpu usage per Container is 200m, but request is 100m. Create a Pod that does not specify any CPU request or limitHere's a manifest for a Pod that has one container. The container does not specify a CPU request, nor does it specify a CPU limit.admin/resource/cpu-constraints-pod-4.yaml apiVersion: v1 kind: Pod metadata: name: constraints-cpu-demo-4 spec: containers: - name: constraints-cpu-demo-4-ctr image: vish/stress Create the Pod:kubectl apply -f https://k8s.io/examples/admin/resource/cpu-constraints-pod-4.yaml --namespace=constraints-cpu-example View detailed information about the Pod:kubectl get pod constraints-cpu-demo-4 --namespace=constraints-cpu-example --output=yaml The output shows that the Pod's single container has a CPU request of 800 millicpu and a CPU limit of 800 millicpu. How did that container get those values?resources: limits: cpu: 800m requests: cpu: 800m Because that container did not specify its own CPU request and limit, the control plane applied the default CPU request and limit from the LimitRange for this namespace.At this point, your Pod may or may not be running. Recall that a prerequisite for this task is that your Nodes must have at least 1 CPU available for use. If each of your Nodes has only 1 CPU, then there might not be enough allocatable CPU on any Node to accommodate a request of 800 millicpu. If you happen to be using Nodes with 2 CPU, then you probably have enough CPU to accommodate the 800 millicpu request.Delete your Pod:kubectl delete pod constraints-cpu-demo-4 --namespace=constraints-cpu-example Enforcement of minimum and maximum CPU constraintsThe maximum and minimum CPU constraints imposed on a namespace by a LimitRange are enforced only when a Pod is created or updated. If you change the LimitRange, it does not affect Pods that were created previously.Motivation for minimum and maximum CPU constraintsAs a cluster administrator, you might want to impose restrictions on the CPU resources that Pods can use. For example:Each Node in a cluster has 2 CPU. You do not want to accept any Pod that requests more than 2 CPU, because no Node in the cluster can support the request.A cluster is shared by your production and development departments. You want to allow production workloads to consume up to 3 CPU, but you want development workloads to be limited to 1 CPU. You create separate namespaces for production and development, and you apply CPU constraints to each namespace.Clean upDelete your namespace:kubectl delete namespace constraints-cpu-example What's nextFor cluster administratorsConfigure Default Memory Requests and Limits for a NamespaceConfigure Default CPU Requests and Limits for a NamespaceConfigure Minimum and Maximum Memory Constraints for a NamespaceConfigure Memory and CPU Quotas for a NamespaceConfigure a Pod Quota for a NamespaceConfigure Quotas for API ObjectsFor app developersAssign Memory Resources to Containers and PodsAssign CPU Resources to Containers and PodsAssign Pod-level CPU and memory resourcesConfigure Quality of Service for Pods


Horizontal Pod AutoscalingIn Kubernetes, a HorizontalPodAutoscaler automatically updates a workload resource (such as a Deployment or StatefulSet), with the aim of automatically scaling the workload to match demand.Horizontal scaling means that the response to increased load is to deploy more Pods. This is different from vertical scaling, which for Kubernetes would mean assigning more resources (for example: memory or CPU) to the Pods that are already running for the workload.If the load decreases, and the number of Pods is above the configured minimum, the HorizontalPodAutoscaler instructs the workload resource (the Deployment, StatefulSet, or other similar resource) to scale back down.Horizontal pod autoscaling does not apply to objects that can't be scaled (for example: a DaemonSet.)The HorizontalPodAutoscaler is implemented as a Kubernetes API resource and a controller. The resource determines the behavior of the controller. The horizontal pod autoscaling controller, running within the Kubernetes control plane, periodically adjusts the desired scale of its target (for example, a Deployment) to match observed metrics such as average CPU utilization, average memory utilization, or any other custom metric you specify.There is walkthrough example of using horizontal pod autoscaling.How does a HorizontalPodAutoscaler work?graph BT hpa[Horizontal Pod Autoscaler] --> scale[Scale] subgraph rc[RC / Deployment] scale end scale -.-> pod1[Pod 1] scale -.-> pod2[Pod 2] scale -.-> pod3[Pod N] classDef hpa fill:#D5A6BD,stroke:#1E1E1D,stroke-width:1px,color:#1E1E1D; classDef rc fill:#F9CB9C,stroke:#1E1E1D,stroke-width:1px,color:#1E1E1D; classDef scale fill:#B6D7A8,stroke:#1E1E1D,stroke-width:1px,color:#1E1E1D; classDef pod fill:#9FC5E8,stroke:#1E1E1D,stroke-width:1px,color:#1E1E1D; class hpa hpa; class rc rc; class scale scale; class pod1,pod2,pod3 podJavaScript must be enabled to view this contentFigure 1. HorizontalPodAutoscaler controls the scale of a Deployment and its ReplicaSetKubernetes implements horizontal pod autoscaling as a control loop that runs intermittently (it is not a continuous process). The interval is set by the --horizontal-pod-autoscaler-sync-period parameter to the kube-controller-manager (and the default interval is 15 seconds).Once during each period, the controller manager queries the resource utilization against the metrics specified in each HorizontalPodAutoscaler definition. The controller manager finds the target resource defined by the scaleTargetRef, then selects the pods based on the target resource's .spec.selector labels, and obtains the metrics from either the resource metrics API (for per-pod resource metrics), or the custom metrics API (for all other metrics).For per-pod resource metrics (like CPU), the controller fetches the metrics from the resource metrics API for each Pod targeted by the HorizontalPodAutoscaler. Then, if a target utilization value is set, the controller calculates the utilization value as a percentage of the equivalent resource request on the containers in each Pod. If a target raw value is set, the raw metric values are used directly. The controller then takes the mean of the utilization or the raw value (depending on the type of target specified) across all targeted Pods, and produces a ratio used to scale the number of desired replicas.Please note that if some of the Pod's containers do not have the relevant resource request set, CPU utilization for the Pod will not be defined and the autoscaler will not take any action for that metric. See the algorithm details section below for more information about how the autoscaling algorithm works.For per-pod custom metrics, the controller functions similarly to per-pod resource metrics, except that it works with raw values, not utilization values.For object metrics and external metrics, a single metric is fetched, which describes the object in question. This metric is compared to the target value, to produce a ratio as above. In the autoscaling/v2 API version, this value can optionally be divided by the number of Pods before the comparison is made.The common use for HorizontalPodAutoscaler is to configure it to fetch metrics from aggregated APIs (metrics.k8s.io, custom.metrics.k8s.io, or external.metrics.k8s.io). The metrics.k8s.io API is usually provided by an add-on named Metrics Server, which needs to be launched separately. For more information about resource metrics, see Metrics Server.Support for metrics APIs explains the stability guarantees and support status for these different APIs.The HorizontalPodAutoscaler controller accesses corresponding workload resources that support scaling (such as Deployments and StatefulSet). These resources each have a subresource named scale, an interface that allows you to dynamically set the number of replicas and examine each of their current states. For general information about subresources in the Kubernetes API, see Kubernetes API Concepts.Algorithm detailsFrom the most basic perspective, the HorizontalPodAutoscaler controller operates on the ratio between desired metric value and current metric value:$$\begin{equation*} desiredReplicas = ceil\left\lceil currentReplicas \times \frac{currentMetricValue}{desiredMetricValue} \right\rceil \end{equation*}$$For example, if the current metric value is 200m, and the desired value is 100m, the number of replicas will be doubled, since \( { 200.0 \div 100.0 } = 2.0 \).If the current value is instead 50m, you'll halve the number of replicas, since \( { 50.0 \div 100.0 } = 0.5 \). The control plane skips any scaling action if the ratio is sufficiently close to 1.0 (within a configurable tolerance, 0.1 by default).When a targetAverageValue or targetAverageUtilization is specified, the currentMetricValue is computed by taking the average of the given metric across all Pods in the HorizontalPodAutoscaler's scale target.Before checking the tolerance and deciding on the final values, the control plane also considers whether any metrics are missing, and how many Pods are Ready. All Pods with a deletion timestamp set (objects with a deletion timestamp are in the process of being shut down / removed) are ignored, and all failed Pods are discarded.If a particular Pod is missing metrics, it is set aside for later; Pods with missing metrics will be used to adjust the final scaling amount.When scaling on CPU, if any pod has yet to become ready (it's still initializing, or possibly is unhealthy) or the most recent metric point for the pod was before it became ready, that pod is set aside as well.Due to technical constraints, the HorizontalPodAutoscaler controller cannot exactly determine the first time a pod becomes ready when determining whether to set aside certain CPU metrics. Instead, it considers a Pod "not yet ready" if it's unready and transitioned to ready within a short, configurable window of time since it started. This value is configured with the --horizontal-pod-autoscaler-initial-readiness-delay flag, and its default is 30 seconds. Once a pod has become ready, it considers any transition to ready to be the first if it occurred within a longer, configurable time since it started. This value is configured with the --horizontal-pod-autoscaler-cpu-initialization-period flag, and its default is 5 minutes.The \( currentMetricValue \over desiredMetricValue \) base scale ratio is then calculated, using the remaining pods not set aside or discarded from above.If there were any missing metrics, the control plane recomputes the average more conservatively, assuming those pods were consuming 100% of the desired value in case of a scale down, and 0% in case of a scale up. This dampens the magnitude of any potential scale.Furthermore, if any not-yet-ready pods were present, and the workload would have scaled up without factoring in missing metrics or not-yet-ready pods, the controller conservatively assumes that the not-yet-ready pods are consuming 0% of the desired metric, further dampening the magnitude of a scale up.After factoring in the not-yet-ready pods and missing metrics, the controller recalculates the usage ratio. If the new ratio reverses the scale direction, or is within the tolerance, the controller doesn't take any scaling action. In other cases, the new ratio is used to decide any change to the number of Pods.Note that the original value for the average utilization is reported back via the HorizontalPodAutoscaler status, without factoring in the not-yet-ready pods or missing metrics, even when the new usage ratio is used.If multiple metrics are specified in a HorizontalPodAutoscaler, this calculation is done for each metric, and then the largest of the desired replica counts is chosen. If any of these metrics cannot be converted into a desired replica count (e.g. due to an error fetching the metrics from the metrics APIs) and a scale down is suggested by the metrics which can be fetched, scaling is skipped. This means that the HPA is still capable of scaling up if one or more metrics give a desiredReplicas greater than the current value.Finally, right before HPA scales the target, the scale recommendation is recorded. The controller considers all recommendations within a configurable window choosing the highest recommendation from within that window. This value can be configured using the --horizontal-pod-autoscaler-downscale-stabilization flag, which defaults to 5 minutes. This means that scaledowns will occur gradually, smoothing out the impact of rapidly fluctuating metric values.Pod readiness and autoscaling metricsThe HorizontalPodAutoscaler (HPA) controller includes two flags that influence how CPU metrics are collected from Pods during startup:--horizontal-pod-autoscaler-cpu-initialization-period (default: 5 minutes)This defines the time window after a Pod starts during which its CPU usage is ignored unless: - The Pod is in a Ready state and - The metric sample was taken entirely during the period it was Ready.This flag helps exclude misleading high CPU usage from initializing Pods (e.g., Java apps warming up) in HPA scaling decisions.--horizontal-pod-autoscaler-initial-readiness-delay (default: 30 seconds)This defines a short delay period after a Pod starts during which the HPA controller treats Pods that are currently Unready as still initializing, even if they have previously transitioned to Ready briefly.It is designed to: - Avoid including Pods that rapidly fluctuate between Ready and Unready during startup. - Ensure stability in the initial readiness signal before HPA considers their metrics valid.Key behaviors:If a Pod is Ready and remains Ready, it can be counted as contributing metrics even within the delay.If a Pod rapidly toggles between Ready and Unready, metrics are ignored until it’s considered stably Ready.Best Practice:If your Pod has a startup phase with high CPU usage:Configure a startupProbe that doesn't pass until the high CPU usage has passed, orEnsure your readinessProbe only reports Ready after the CPU spike subsides, using initialDelaySeconds.And ideally also set --horizontal-pod-autoscaler-cpu-initialization-period to cover the startup duration.API ObjectThe Horizontal Pod Autoscaler is an API resource in the Kubernetes autoscaling API group. The current stable version can be found in the autoscaling/v2 API version which includes support for scaling on memory and custom metrics. The new fields introduced in autoscaling/v2 are preserved as annotations when working with autoscaling/v1.When you create a HorizontalPodAutoscaler API object, make sure the name specified is a valid DNS subdomain name. More details about the API object can be found at HorizontalPodAutoscaler Object.Stability of workload scaleWhen managing the scale of a group of replicas using the HorizontalPodAutoscaler, it is possible that the number of replicas keeps fluctuating frequently due to the dynamic nature of the metrics evaluated. This is sometimes referred to as thrashing, or flapping. It's similar to the concept of hysteresis in cybernetics.Autoscaling during rolling updateKubernetes lets you perform a rolling update on a Deployment. In that case, the Deployment manages the underlying ReplicaSets for you. When you configure autoscaling for a Deployment, you bind a HorizontalPodAutoscaler to a single Deployment. The HorizontalPodAutoscaler manages the replicas field of the Deployment. The deployment controller is responsible for setting the replicas of the underlying ReplicaSets so that they add up to a suitable number during the rollout and also afterwards.If you perform a rolling update of a StatefulSet that has an autoscaled number of replicas, the StatefulSet directly manages its set of Pods (there is no intermediate resource similar to ReplicaSet).Support for resource metricsAny HPA target can be scaled based on the resource usage of the pods in the scaling target. When defining the pod specification the resource requests like cpu and memory should be specified. This is used to determine the resource utilization and used by the HPA controller to scale the target up or down. To use resource utilization based scaling specify a metric source like this:type: Resource resource: name: cpu target: type: Utilization averageUtilization: 60 With this metric the HPA controller will keep the average utilization of the pods in the scaling target at 60%. Utilization is the ratio between the current usage of resource to the requested resources of the pod. See Algorithm for more details about how the utilization is calculated and averaged.Note:Since the resource usages of all the containers are summed up the total pod utilization may not accurately represent the individual container resource usage. This could lead to situations where a single container might be running with high usage and the HPA will not scale out because the overall pod usage is still within acceptable limits.Container resource metricsFEATURE STATE: Kubernetes v1.30 [stable] (enabled by default: true)The HorizontalPodAutoscaler API also supports a container metric source where the HPA can track the resource usage of individual containers across a set of Pods, in order to scale the target resource. This lets you configure scaling thresholds for the containers that matter most in a particular Pod. For example, if you have a web application and a sidecar container that provides logging, you can scale based on the resource use of the web application, ignoring the sidecar container and its resource use.If you revise the target resource to have a new Pod specification with a different set of containers, you should revise the HPA spec if that newly added container should also be used for scaling. If the specified container in the metric source is not present or only present in a subset of the pods then those pods are ignored and the recommendation is recalculated. See Algorithm for more details about the calculation. To use container resources for autoscaling define a metric source as follows:type: ContainerResource containerResource: name: cpu container: application target: type: Utilization averageUtilization: 60 In the above example the HPA controller scales the target such that the average utilization of the cpu in the application container of all the pods is 60%.Note:If you change the name of a container that a HorizontalPodAutoscaler is tracking, you can make that change in a specific order to ensure scaling remains available and effective whilst the change is being applied. Before you update the resource that defines the container (such as a Deployment), you should update the associated HPA to track both the new and old container names. This way, the HPA is able to calculate a scaling recommendation throughout the update process.Once you have rolled out the container name change to the workload resource, tidy up by removing the old container name from the HPA specification.Scaling on custom metricsFEATURE STATE: Kubernetes v1.23 [stable](the autoscaling/v2beta2 API version previously provided this ability as a beta feature)Provided that you use the autoscaling/v2 API version, you can configure a HorizontalPodAutoscaler to scale based on a custom metric (that is not built in to Kubernetes or any Kubernetes component). The HorizontalPodAutoscaler controller then queries for these custom metrics from the Kubernetes API.See Support for metrics APIs for the requirements.Scaling on multiple metricsFEATURE STATE: Kubernetes v1.23 [stable](the autoscaling/v2beta2 API version previously provided this ability as a beta feature)Provided that you use the autoscaling/v2 API version, you can specify multiple metrics for a HorizontalPodAutoscaler to scale on. Then, the HorizontalPodAutoscaler controller evaluates each metric, and proposes a new scale based on that metric. The HorizontalPodAutoscaler takes the maximum scale recommended for each metric and sets the workload to that size (provided that this isn't larger than the overall maximum that you configured).Support for metrics APIsBy default, the HorizontalPodAutoscaler controller retrieves metrics from a series of APIs. In order for it to access these APIs, cluster administrators must ensure that:The API aggregation layer is enabled.The corresponding APIs are registered:For resource metrics, this is the metrics.k8s.io API, generally provided by metrics-server. It can be launched as a cluster add-on.For custom metrics, this is the custom.metrics.k8s.io API. It's provided by "adapter" API servers provided by metrics solution vendors. Check with your metrics pipeline to see if there is a Kubernetes metrics adapter available.For external metrics, this is the external.metrics.k8s.io API. It may be provided by the custom metrics adapters provided above.For more information on these different metrics paths and how they differ please see the relevant design proposals for the HPA V2, custom.metrics.k8s.io and external.metrics.k8s.io.For examples of how to use them see the walkthrough for using custom metrics and the walkthrough for using external metrics.Configurable scaling behaviorFEATURE STATE: Kubernetes v1.23 [stable](the autoscaling/v2beta2 API version previously provided this ability as a beta feature)If you use the v2 HorizontalPodAutoscaler API, you can use the behavior field (see the API reference) to configure separate scale-up and scale-down behaviors. You specify these behaviours by setting scaleUp and / or scaleDown under the behavior field.Scaling policies let you control the rate of change of replicas while scaling. Also two settings can be used to prevent flapping: you can specify a stabilization window for smoothing replica counts, and a tolerance to ignore minor metric fluctuations below a specified threshold.Scaling policiesOne or more scaling policies can be specified in the behavior section of the spec. When multiple policies are specified the policy which allows the highest amount of change is the policy which is selected by default. The following example shows this behavior while scaling down:behavior: scaleDown: policies: - type: Pods value: 4 periodSeconds: 60 - type: Percent value: 10 periodSeconds: 60 periodSeconds indicates the length of time in the past for which the policy must hold true. The maximum value that you can set for periodSeconds is 1800 (half an hour). The first policy (Pods) allows at most 4 replicas to be scaled down in one minute. The second policy (Percent) allows at most 10% of the current replicas to be scaled down in one minute.Since by default the policy which allows the highest amount of change is selected, the second policy will only be used when the number of pod replicas is more than 40. With 40 or less replicas, the first policy will be applied. For instance if there are 80 replicas and the target has to be scaled down to 10 replicas then during the first step 8 replicas will be reduced. In the

Horizontal Pod Autoscaling


Windows StorageThis page provides an storage overview specific to the Windows operating system.Persistent storageWindows has a layered filesystem driver to mount container layers and create a copy filesystem based on NTFS. All file paths in the container are resolved only within the context of that container.With Docker, volume mounts can only target a directory in the container, and not an individual file. This limitation does not apply to containerd.Volume mounts cannot project files or directories back to the host filesystem.Read-only filesystems are not supported because write access is always required for the Windows registry and SAM database. However, read-only volumes are supported.Volume user-masks and permissions are not available. Because the SAM is not shared between the host & container, there's no mapping between them. All permissions are resolved within the context of the container.As a result, the following storage functionality is not supported on Windows nodes:Volume subpath mounts: only the entire volume can be mounted in a Windows containerSubpath volume mounting for SecretsHost mount projectionRead-only root filesystem (mapped volumes still support readOnly)Block device mappingMemory as the storage medium (for example, emptyDir.medium set to Memory)File system features like uid/gid; per-user Linux filesystem permissionsSetting secret permissions with DefaultMode (due to UID/GID dependency)NFS based storage/volume supportExpanding the mounted volume (resizefs)Kubernetes volumes enable complex applications, with data persistence and Pod volume sharing requirements, to be deployed on Kubernetes. Management of persistent volumes associated with a specific storage back-end or protocol includes actions such as provisioning/de-provisioning/resizing of volumes, attaching/detaching a volume to/from a Kubernetes node and mounting/dismounting a volume to/from individual containers in a pod that needs to persist data.Volume management components are shipped as Kubernetes volume plugin. The following broad classes of Kubernetes volume plugins are supported on Windows:FlexVolume pluginsPlease note that FlexVolumes have been deprecated as of 1.23CSI PluginsIn-tree volume pluginsThe following in-tree plugins support persistent storage on Windows nodes:azureFilevsphereVolume

Windows Storage