# Node-pressure Eviction

Node-pressure EvictionNode-pressure eviction is the process by which the kubelet proactively terminates pods to reclaim resource on nodes.The kubelet monitors resources like memory, disk space, and filesystem inodes on your cluster's nodes. When one or more of these resources reach specific consumption levels, the kubelet can proactively fail one or more pods on the node to reclaim resources and prevent starvation.During a node-pressure eviction, the kubelet sets the phase for the selected pods to Failed, and terminates the Pod.Node-pressure eviction is not the same as API-initiated eviction.The kubelet does not respect your configured PodDisruptionBudget or the pod's terminationGracePeriodSeconds. If you use soft eviction thresholds, the kubelet respects your configured eviction-max-pod-grace-period. If you use hard eviction thresholds, the kubelet uses a 0s grace period (immediate shutdown) for termination.Self healing behaviorThe kubelet attempts to reclaim node-level resources before it terminates end-user pods. For example, it removes unused container images when disk resources are starved.If the pods are managed by a workload management object (such as StatefulSet or Deployment) that replaces failed pods, the control plane (kube-controller-manager) creates new pods in place of the evicted pods.Self healing for static podsIf you are running a static pod on a node that is under resource pressure, the kubelet may evict that static Pod. The kubelet then tries to create a replacement, because static Pods always represent an intent to run a Pod on that node.The kubelet takes the priority of the static pod into account when creating a replacement. If the static pod manifest specifies a low priority, and there are higher-priority Pods defined within the cluster's control plane, and the node is under resource pressure, the kubelet may not be able to make room for that static pod. The kubelet continues to attempt to run all static pods even when there is resource pressure on a node.Eviction signals and thresholdsThe kubelet uses various parameters to make eviction decisions, like the following:Eviction signalsEviction thresholdsMonitoring intervalsEviction signalsEviction signals are the current state of a particular resource at a specific point in time. The kubelet uses eviction signals to make eviction decisions by comparing the signals to eviction thresholds, which are the minimum amount of the resource that should be available on the node.The kubelet uses the following eviction signals:Eviction SignalDescriptionLinux Onlymemory.availablememory.available := node.status.capacity[memory] - node.stats.memory.workingSetnodefs.availablenodefs.available := node.stats.fs.availablenodefs.inodesFreenodefs.inodesFree := node.stats.fs.inodesFree•imagefs.availableimagefs.available := node.stats.runtime.imagefs.availableimagefs.inodesFreeimagefs.inodesFree := node.stats.runtime.imagefs.inodesFree•containerfs.availablecontainerfs.available := node.stats.runtime.containerfs.availablecontainerfs.inodesFreecontainerfs.inodesFree := node.stats.runtime.containerfs.inodesFree•pid.availablepid.available := node.stats.rlimit.maxpid - node.stats.rlimit.curproc•In this table, the Description column shows how kubelet gets the value of the signal. Each signal supports either a percentage or a literal value. The kubelet calculates the percentage value relative to the total capacity associated with the signal.Memory signalsOn Linux nodes, the value for memory.available is derived from the cgroupfs instead of tools like free -m. This is important because free -m does not work in a container, and if users use the node allocatable feature, out of resource decisions are made local to the end user Pod part of the cgroup hierarchy as well as the root node. This script or cgroupv2 script reproduces the same set of steps that the kubelet performs to calculate memory.available. The kubelet excludes inactive_file (the number of bytes of file-backed memory on the inactive LRU list) from its calculation, as it assumes that memory is reclaimable under pressure.On Windows nodes, the value for memory.available is derived from the node's global memory commit levels (queried through the GetPerformanceInfo() system call) by subtracting the node's global CommitTotal from the node's CommitLimit. Please note that CommitLimit can change if the node's page-file size changes!Filesystem signalsThe kubelet recognizes three specific filesystem identifiers that can be used with eviction signals (<identifier>.inodesFree or <identifier>.available):nodefs: The node's main filesystem, used for local disk volumes, emptyDir volumes not backed by memory, log storage, ephemeral storage, and more. For example, nodefs contains /var/lib/kubelet.imagefs: An optional filesystem that container runtimes can use to store container images (which are the read-only layers) and container writable layers.containerfs: An optional filesystem that container runtime can use to store the writeable layers. Similar to the main filesystem (see nodefs), it's used to store local disk volumes, emptyDir volumes not backed by memory, log storage, and ephemeral storage, except for the container images. When containerfs is used, the imagefs filesystem can be split to only store images (read-only layers) and nothing else.Note:FEATURE STATE: Kubernetes v1.31 [beta] (enabled by default: true)The split image filesystem feature, which enables support for the containerfs filesystem, adds several new eviction signals, thresholds and metrics. To use containerfs, the Kubernetes release v1.34 requires the KubeletSeparateDiskGC feature gate to be enabled. Currently, only CRI-O (v1.29 or higher) offers the containerfs filesystem support.As such, kubelet generally allows three options for container filesystems:Everything is on the single nodefs, also referred to as "rootfs" or simply "root", and there is no dedicated image filesystem.Container storage (see nodefs) is on a dedicated disk, and imagefs (writable and read-only layers) is separate from the root filesystem. This is often referred to as "split disk" (or "separate disk") filesystem.Container filesystem containerfs (same as nodefs plus writable layers) is on root and the container images (read-only layers) are stored on separate imagefs. This is often referred to as "split image" filesystem.The kubelet will attempt to auto-discover these filesystems with their current configuration directly from the underlying container runtime and will ignore other local node filesystems.The kubelet does not support other container filesystems or storage configurations, and it does not currently support multiple filesystems for images and containers.Deprecated kubelet garbage collection featuresSome kubelet garbage collection features are deprecated in favor of eviction:Existing FlagRationale--maximum-dead-containersdeprecated once old logs are stored outside of container's context--maximum-dead-containers-per-containerdeprecated once old logs are stored outside of container's context--minimum-container-ttl-durationdeprecated once old logs are stored outside of container's contextEviction thresholdsYou can specify custom eviction thresholds for the kubelet to use when it makes eviction decisions. You can configure soft and hard eviction thresholds.Eviction thresholds have the form [eviction-signal][operator][quantity], where:eviction-signal is the eviction signal to use.operator is the relational operator you want, such as < (less than).quantity is the eviction threshold amount, such as 1Gi. The value of quantity must match the quantity representation used by Kubernetes. You can use either literal values or percentages (%).For example, if a node has 10GiB of total memory and you want trigger eviction if the available memory falls below 1GiB, you can define the eviction threshold as either memory.available<10% or memory.available<1Gi (you cannot use both).Soft eviction thresholdsA soft eviction threshold pairs an eviction threshold with a required administrator-specified grace period. The kubelet does not evict pods until the grace period is exceeded. The kubelet returns an error on startup if you do not specify a grace period.You can specify both a soft eviction threshold grace period and a maximum allowed pod termination grace period for kubelet to use during evictions. If you specify a maximum allowed grace period and the soft eviction threshold is met, the kubelet uses the lesser of the two grace periods. If you do not specify a maximum allowed grace period, the kubelet kills evicted pods immediately without graceful termination.You can use the following flags to configure soft eviction thresholds:eviction-soft: A set of eviction thresholds like memory.available<1.5Gi that can trigger pod eviction if held over the specified grace period.eviction-soft-grace-period: A set of eviction grace periods like memory.available=1m30s that define how long a soft eviction threshold must hold before triggering a Pod eviction.eviction-max-pod-grace-period: The maximum allowed grace period (in seconds) to use when terminating pods in response to a soft eviction threshold being met.Hard eviction thresholdsA hard eviction threshold has no grace period. When a hard eviction threshold is met, the kubelet kills pods immediately without graceful termination to reclaim the starved resource.You can use the eviction-hard flag to configure a set of hard eviction thresholds like memory.available<1Gi.The kubelet has the following default hard eviction thresholds:memory.available<100Mi (Linux nodes)memory.available<500Mi (Windows nodes)nodefs.available<10%imagefs.available<15%nodefs.inodesFree<5% (Linux nodes)imagefs.inodesFree<5% (Linux nodes)These default values of hard eviction thresholds will only be set if none of the parameters is changed. If you change the value of any parameter, then the values of other parameters will not be inherited as the default values and will be set to zero. In order to provide custom values, you should provide all the thresholds respectively. You can also set the kubelet config MergeDefaultEvictionSettings to true in the kubelet configuration file. If set to true and any parameter is changed, then the other parameters will inherit their default values instead of 0.The containerfs.available and containerfs.inodesFree (Linux nodes) default eviction thresholds will be set as follows:If a single filesystem is used for everything, then containerfs thresholds are set the same as nodefs.If separate filesystems are configured for both images and containers, then containerfs thresholds are set the same as imagefs.Setting custom overrides for thresholds related to containersfs is currently not supported, and a warning will be issued if an attempt to do so is made; any provided custom values will, as such, be ignored.Eviction monitoring intervalThe kubelet evaluates eviction thresholds based on its configured housekeeping-interval, which defaults to 10s.Node conditionsThe kubelet reports node conditions to reflect that the node is under pressure because hard or soft eviction threshold is met, independent of configured grace periods.The kubelet maps eviction signals to node conditions as follows:Node ConditionEviction SignalDescriptionMemoryPressurememory.availableAvailable memory on the node has satisfied an eviction thresholdDiskPressurenodefs.available, nodefs.inodesFree, imagefs.available, imagefs.inodesFree, containerfs.available, or containerfs.inodesFreeAvailable disk space and inodes on either the node's root filesystem, image filesystem, or container filesystem has satisfied an eviction thresholdPIDPressurepid.availableAvailable processes identifiers on the (Linux) node has fallen below an eviction thresholdThe control plane also maps these node conditions to taints.The kubelet updates the node conditions based on the configured --node-status-update-frequency, which defaults to 10s.Node condition oscillationIn some cases, nodes oscillate above and below soft eviction thresholds without holding for the defined grace periods. This causes the reported node condition to constantly switch between true and false, leading to bad eviction decisions.To protect against oscillation, you can use the eviction-pressure-transition-period flag, which controls how long the kubelet must wait before transitioning a node condition to a different state. The transition period has a default value of 5m.Reclaiming node level resourcesThe kubelet tries to reclaim node-level resources before it evicts end-user pods.When a DiskPressure node condition is reported, the kubelet reclaims node-level resources based on the filesystems on the node.Without imagefs or containerfsIf the node only has a nodefs filesystem that meets eviction thresholds, the kubelet frees up disk space in the following order:Garbage collect dead pods and containers.Delete unused images.With imagefsIf the node has a dedicated imagefs filesystem for container runtimes to use, the kubelet does the following:If the nodefs filesystem meets the eviction thresholds, the kubelet garbage collects dead pods and containers.If the imagefs filesystem meets the eviction thresholds, the kubelet deletes all unused images.With imagefs and containerfsIf the node has a dedicated containerfs alongside the imagefs filesystem configured for the container runtimes to use, then kubelet will attempt to reclaim resources as follows:If the containerfs filesystem meets the eviction thresholds, the kubelet garbage collects dead pods and containers.If the imagefs filesystem meets the eviction thresholds, the kubelet deletes all unused images.Pod selection for kubelet evictionIf the kubelet's attempts to reclaim node-level resources don't bring the eviction signal below the threshold, the kubelet begins to evict end-user pods.The kubelet uses the following parameters to determine the pod eviction order:Whether the pod's resource usage exceeds requestsPod PriorityThe pod's resource usage relative to requestsAs a result, kubelet ranks and evicts pods in the following order:BestEffort or Burstable pods where the usage exceeds requests. These pods are evicted based on their Priority and then by how much their usage level exceeds the request.Guaranteed pods and Burstable pods where the usage is less than requests are evicted last, based on their Priority.Note:The kubelet does not use the pod's QoS class to determine the eviction order. You can use the QoS class to estimate the most likely pod eviction order when reclaiming resources like memory. QoS classification does not apply to EphemeralStorage requests, so the above scenario will not apply if the node is, for example, under DiskPressure.Guaranteed pods are guaranteed only when requests and limits are specified for all the containers and they are equal. These pods will never be evicted because of another pod's resource consumption. If a system daemon (such as kubelet and journald) is consuming more resources than were reserved via system-reserved or kube-reserved allocations, and the node only has Guaranteed or Burstable pods using less resources than requests left on it, then the kubelet must choose to evict one of these pods to preserve node stability and to limit the impact of resource starvation on other pods. In this case, it will choose to evict pods of lowest Priority first.If you are running a static pod and want to avoid having it evicted under resource pressure, set the priority field for that Pod directly. Static pods do not support the priorityClassName field.When the kubelet evicts pods in response to inode or process ID starvation, it uses the Pods' relative priority to determine the eviction order, because inodes and PIDs have no requests.The kubelet sorts pods differently based on whether the node has a dedicated imagefs or containerfs filesystem:Without imagefs or containerfs (nodefs and imagefs use the same filesystem)If nodefs triggers evictions, the kubelet sorts pods based on their total disk usage (local volumes + logs and a writable layer of all containers).With imagefs (nodefs and imagefs filesystems are separate)If nodefs triggers evictions, the kubelet sorts pods based on nodefs usage (local volumes + logs of all containers).If imagefs triggers evictions, the kubelet sorts pods based on the writable layer usage of all containers.With imagesfs and containerfs (imagefs and containerfs have been split)If containerfs triggers evictions, the kubelet sorts pods based on containerfs usage (local volumes + logs and a writable layer of all containers).If imagefs triggers evictions, the kubelet sorts pods based on the storage of images rank, which represents the disk usage of a given image.Minimum eviction reclaimNote:As of Kubernetes v1.34, you cannot set a custom value for the containerfs.available metric. The configuration for this specific metric will be set automatically to reflect values set for either the nodefs or imagefs, depending on the configuration.In some cases, pod eviction only reclaims a small amount of the starved resource. This can lead to the kubelet repeatedly hitting the configured eviction thresholds and triggering multiple evictions.You can use the --eviction-minimum-reclaim flag or a kubelet config file to configure a minimum reclaim amount for each resource. When the kubelet notices that a resource is starved, it continues to reclaim that resource until it reclaims the quantity you specify.For example, the following configuration sets minimum reclaim amounts:apiVersion: kubelet.config.k8s.io/v1beta1 kind: KubeletConfiguration evictionHard: memory.available: "500Mi" nodefs.available: "1Gi" imagefs.available: "100Gi" evictionMinimumReclaim: memory.available: "0Mi" nodefs.available: "500Mi" imagefs.available: "2Gi" In this example, if the nodefs.available signal meets the eviction threshold, the kubelet reclaims the resource until the signal reaches the threshold of 1GiB, and then continues to reclaim the minimum amount of 500MiB, until the available nodefs storage value reaches 1.5GiB.Similarly, the kubelet tries to reclaim the imagefs resource until the imagefs.available value reaches 102Gi, representing 102 GiB of available container image storage. If the amount of storage that the kubelet could reclaim is less than 2GiB, the kubelet doesn't reclaim anything.The default eviction-minimum-reclaim is 0 for all resources.Node out of memory behaviorIf the node experiences an out of memory (OOM) event prior to the kubelet being able to reclaim memory, the node depends on the oom_killer to respond.The kubelet sets an oom_score_adj value for each container based on the QoS for the pod.Quality of Serviceoom_score_adjGuaranteed-997BestEffort1000Burstablemin(max(2, 1000 - (1000 × memoryRequestBytes) / machineMemoryCapacityBytes), 999)Note:The kubelet also sets an oom_score_adj value of -997 for any containers in Pods that have system-node-critical Priority.If the kubelet can't reclaim memory before a node experiences OOM, the oom_killer calculates an oom_score based on the percentage of memory it's using on the node, and then adds the oom_score_adj to get an effective oom_score for each container. It then kills the container with the highest score.This means that containers in low QoS pods that consume a large amount of memory relative to their scheduling requests are killed first.Unlike pod eviction, if a container is OOM killed, the kubelet can restart it based on its restartPolicy.Good practicesThe following sections describe good practice for eviction configuration.Schedulable resources and eviction policiesWhen you configure the kubelet with an eviction policy, you should make sure that the scheduler will not schedule pods if they will trigger eviction because they immediately induce memory pressure.Consider the following scenario:Node memory capacity: 10GiBOperator wants to reserve 10% of memory capacity for system daemons (kernel, kubelet, etc.)Operator wants to evict Pods at 95% memory utilization to reduce incidence of system OOM.For this to work, the kubelet is launched as follows:--eviction-hard=memory.available<500Mi --system-reserved=memory=1.5Gi In this configuration, the --system-reserved flag reserves 1.5GiB of memory for the system, which is 10% of the total memory + the eviction threshold amount.The node can reach the eviction threshold if a pod is using more than its request, or if the system is using more than 1GiB of memory, which makes the memory.available signal fall below 500MiB and triggers the threshold.DaemonSets and node-pressure evictionPod priority is a major factor in making eviction decisions. If you do not want the kubelet to evict pods that belong to a DaemonSet, give those pods a high enough priority by specifying a suitable priorityClassName in the pod spec. You can also use a lower priority, or the default, to only allow pods from that DaemonSet to run when there are enough resources.Known issuesThe following sections describe known issues related to out of resource handling.kubelet may not observe memory pressure right awayBy default, the kubelet polls cAdvisor to collect memory usage stats at a regular interval. If memory usage increases within that window rapidly, the kubelet may not observe MemoryPressure fast enough, and the OOM killer will still be invoked.You can use the --kernel-memcg-notification flag to enable the memcg notification API on the kubelet to get notified immediately when a threshold is crossed.If you are not trying to achieve extreme utilization, but a sensible measure of overcommit, a viable workaround for this issue is to use the --kube-reserved and --system-reserved flags to allocate memory for the system.active_file memory is not considered as available memoryOn Linux, the kernel tracks the number of bytes of file-backed memory on active least recently used (LRU) list as the active_file statistic. The kubelet treats active_file memory areas as not reclaimable. For workloads that make intensive use of block-backed local storage, including ephemeral local storage, kernel-level caches of file and block data means that many recently accessed cache pages are likely to be counted as active_file. If enough of these kernel block buffers are on the active LRU list, the kubelet is liable to observe this as high resource use and taint the node as experiencing memory pressure - triggering pod eviction.For more details, see https://github.com/kubernetes/kubernetes/issues/43916You can work around that behavior by setting the memory limit and memory request the same for containers likely to perform intensive I/O activity. You will need to estimate or measure an optimal memory limit value for that container.What's nextLearn about API-initiated EvictionLearn about Pod Priority and PreemptionLearn about PodDisruptionBudgetsLearn about Quality of Service (QoS)Check out the Eviction API


# kubectl logs

kubectl logsSynopsisPrint the logs for a container in a pod or specified resource. If the pod has only one container, the container name is optional.kubectl logs [-f] [-p] (POD | TYPE/NAME) [-c CONTAINER] Examples # Return snapshot logs from pod nginx with only one container kubectl logs nginx # Return snapshot logs from pod nginx, prefixing each line with the source pod and container name kubectl logs nginx --prefix # Return snapshot logs from pod nginx, limiting output to 500 bytes kubectl logs nginx --limit-bytes=500 # Return snapshot logs from pod nginx, waiting up to 20 seconds for it to start running. kubectl logs nginx --pod-running-timeout=20s # Return snapshot logs from pod nginx with multi containers kubectl logs nginx --all-containers=true # Return snapshot logs from all pods in the deployment nginx kubectl logs deployment/nginx --all-pods=true # Return snapshot logs from all containers in pods defined by label app=nginx kubectl logs -l app=nginx --all-containers=true # Return snapshot logs from all pods defined by label app=nginx, limiting concurrent log requests to 10 pods kubectl logs -l app=nginx --max-log-requests=10 # Return snapshot of previous terminated ruby container logs from pod web-1 kubectl logs -p -c ruby web-1 # Begin streaming the logs from pod nginx, continuing even if errors occur kubectl logs nginx -f --ignore-errors=true # Begin streaming the logs of the ruby container in pod web-1 kubectl logs -f -c ruby web-1 # Begin streaming the logs from all containers in pods defined by label app=nginx kubectl logs -f -l app=nginx --all-containers=true # Display only the most recent 20 lines of output in pod nginx kubectl logs --tail=20 nginx # Show all logs from pod nginx written in the last hour kubectl logs --since=1h nginx # Show all logs with timestamps from pod nginx starting from August 30, 2024, at 06:00:00 UTC kubectl logs nginx --since-time=2024-08-30T06:00:00Z --timestamps=true # Show logs from a kubelet with an expired serving certificate kubectl logs --insecure-skip-tls-verify-backend nginx # Return snapshot logs from first container of a job named hello kubectl logs job/hello # Return snapshot logs from container nginx-1 of a deployment named nginx kubectl logs deployment/nginx -c nginx-1 Options--all-containersGet all containers' logs in the pod(s).--all-podsGet logs from all pod(s). Sets prefix to true.-c, --container stringPrint the logs of this container-f, --followSpecify if the logs should be streamed.-h, --helphelp for logs--ignore-errorsIf watching / following pod logs, allow for any errors that occur to be non-fatal--insecure-skip-tls-verify-backendSkip verifying the identity of the kubelet that logs are requested from. In theory, an attacker could provide invalid log content back. You might want to use this if your kubelet serving certificates have expired.--limit-bytes intMaximum bytes of logs to return. Defaults to no limit.--max-log-requests int Default: 5Specify maximum number of concurrent logs to follow when using by a selector. Defaults to 5.--pod-running-timeout duration Default: 20sThe length of time (like 5s, 2m, or 3h, higher than zero) to wait until at least one pod is running--prefixPrefix each log line with the log source (pod name and container name)-p, --previousIf true, print the logs for the previous instance of the container in a pod if it exists.-l, --selector stringSelector (label query) to filter on, supports '=', '==', '!=', 'in', 'notin'.(e.g. -l key1=value1,key2=value2,key3 in (value3)). Matching objects must satisfy all of the specified label constraints.--since durationOnly return logs newer than a relative duration like 5s, 2m, or 3h. Defaults to all logs. Only one of since-time / since may be used.--since-time stringOnly return logs after a specific date (RFC3339). Defaults to all logs. Only one of since-time / since may be used.--tail int Default: -1Lines of recent log file to display. Defaults to -1 with no selector, showing all log lines otherwise 10, if a selector is provided.--timestampsInclude timestamps on each line in the log outputParent Options Inherited--as stringUsername to impersonate for the operation. User could be a regular user or a service account in a namespace.--as-group stringsGroup to impersonate for the operation, this flag can be repeated to specify multiple groups.--as-uid stringUID to impersonate for the operation.--cache-dir string Default: "$HOME/.kube/cache"Default cache directory--certificate-authority stringPath to a cert file for the certificate authority--client-certificate stringPath to a client certificate file for TLS--client-key stringPath to a client key file for TLS--cluster stringThe name of the kubeconfig cluster to use--context stringThe name of the kubeconfig context to use--disable-compressionIf true, opt-out of response compression for all requests to the server--insecure-skip-tls-verifyIf true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure--kubeconfig stringPath to the kubeconfig file to use for CLI requests.--kuberc stringPath to the kuberc file to use for preferences. This can be disabled by exporting KUBECTL_KUBERC=false feature gate or turning off the feature KUBERC=off.--match-server-versionRequire server version to match client version-n, --namespace stringIf present, the namespace scope for this CLI request--password stringPassword for basic authentication to the API server--profile string Default: "none"Name of profile to capture. One of (none|cpu|heap|goroutine|threadcreate|block|mutex)--profile-output string Default: "profile.pprof"Name of the file to write the profile to--request-timeout string Default: "0"The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests.-s, --server stringThe address and port of the Kubernetes API server--storage-driver-buffer-duration duration Default: 1m0sWrites in the storage driver will be buffered for this duration, and committed to the non memory backends as a single transaction--storage-driver-db string Default: "cadvisor"database name--storage-driver-host string Default: "localhost:8086"database host:port--storage-driver-password string Default: "root"database password--storage-driver-secureuse secure connection with database--storage-driver-table string Default: "stats"table name--storage-driver-user string Default: "root"database username--tls-server-name stringServer name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used--token stringBearer token for authentication to the API server--user stringThe name of the kubeconfig user to use--username stringUsername for basic authentication to the API server--version version[=true]--version, --version=raw prints version information and quits; --version=vX.Y.Z... sets the reported version--warnings-as-errorsTreat warnings received from the server as errors and exit with a non-zero exit codeSee Alsokubectl - kubectl controls the Kubernetes cluster managerThis page is automatically generated.If you plan to report an issue with this page, mention that the page is auto-generated in your issue description. The fix may need to happen elsewhere in the Kubernetes project.


# StatefulSet Basics

StatefulSet BasicsThis tutorial provides an introduction to managing applications with StatefulSets. It demonstrates how to create, delete, scale, and update the Pods of StatefulSets.Before you beginBefore you begin this tutorial, you should familiarize yourself with the following Kubernetes concepts:PodsCluster DNSHeadless ServicesPersistentVolumesPersistentVolume ProvisioningThe kubectl command line toolYou need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. It is recommended to run this tutorial on a cluster with at least two nodes that are not acting as control plane hosts. If you do not already have a cluster, you can create one by using minikube or you can use one of these Kubernetes playgrounds:iximiuz LabsKillercodaKodeKloudPlay with KubernetesYou should configure kubectl to use a context that uses the default namespace. If you are using an existing cluster, make sure that it's OK to use that cluster's default namespace to practice. Ideally, practice in a cluster that doesn't run any real workloads.It's also useful to read the concept page about StatefulSets.Note:This tutorial assumes that your cluster is configured to dynamically provision PersistentVolumes. You'll also need to have a default StorageClass. If your cluster is not configured to provision storage dynamically, you will have to manually provision two 1 GiB volumes prior to starting this tutorial and set up your cluster so that those PersistentVolumes map to the PersistentVolumeClaim templates that the StatefulSet defines.ObjectivesStatefulSets are intended to be used with stateful applications and distributed systems. However, the administration of stateful applications and distributed systems on Kubernetes is a broad, complex topic. In order to demonstrate the basic features of a StatefulSet, and not to conflate the former topic with the latter, you will deploy a simple web application using a StatefulSet.After this tutorial, you will be familiar with the following.How to create a StatefulSetHow a StatefulSet manages its PodsHow to delete a StatefulSetHow to scale a StatefulSetHow to update a StatefulSet's PodsCreating a StatefulSetBegin by creating a StatefulSet (and the Service that it relies upon) using the example below. It is similar to the example presented in the StatefulSets concept. It creates a headless Service, nginx, to publish the IP addresses of Pods in the StatefulSet, web.application/web/web.yaml apiVersion: v1 kind: Service metadata: name: nginx labels: app: nginx spec: ports: - port: 80 name: web clusterIP: None selector: app: nginx --- apiVersion: apps/v1 kind: StatefulSet metadata: name: web spec: serviceName: "nginx" replicas: 2 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - name: nginx image: registry.k8s.io/nginx-slim:0.21 ports: - containerPort: 80 name: web volumeMounts: - name: www mountPath: /usr/share/nginx/html volumeClaimTemplates: - metadata: name: www spec: accessModes: [ "ReadWriteOnce" ] resources: requests: storage: 1Gi You will need to use at least two terminal windows. In the first terminal, use kubectl get to watch the creation of the StatefulSet's Pods.# use this terminal to run commands that specify --watch # end this watch when you are asked to start a new watch kubectl get pods --watch -l app=nginx In the second terminal, use kubectl apply to create the headless Service and StatefulSet:kubectl apply -f https://k8s.io/examples/application/web/web.yaml service/nginx created statefulset.apps/web created The command above creates two Pods, each running an NGINX webserver. Get the nginx Service...kubectl get service nginx NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE nginx ClusterIP None <none> 80/TCP 12s ...then get the web StatefulSet, to verify that both were created successfully:kubectl get statefulset web NAME READY AGE web 2/2 37s Ordered Pod creationA StatefulSet defaults to creating its Pods in a strict order.For a StatefulSet with n replicas, when Pods are being deployed, they are created sequentially, ordered from {0..n-1}. Examine the output of the kubectl get command in the first terminal. Eventually, the output will look like the example below.# Do not start a new watch; # this should already be running kubectl get pods --watch -l app=nginx NAME READY STATUS RESTARTS AGE web-0 0/1 Pending 0 0s web-0 0/1 Pending 0 0s web-0 0/1 ContainerCreating 0 0s web-0 1/1 Running 0 19s web-1 0/1 Pending 0 0s web-1 0/1 Pending 0 0s web-1 0/1 ContainerCreating 0 0s web-1 1/1 Running 0 18s Notice that the web-1 Pod is not launched until the web-0 Pod is Running (see Pod Phase) and Ready (see type in Pod Conditions).Later in this tutorial you will practice parallel startup.Note:To configure the integer ordinal assigned to each Pod in a StatefulSet, see Start ordinal.Pods in a StatefulSetPods in a StatefulSet have a unique ordinal index and a stable network identity.Examining the Pod's ordinal indexGet the StatefulSet's Pods:kubectl get pods -l app=nginx NAME READY STATUS RESTARTS AGE web-0 1/1 Running 0 1m web-1 1/1 Running 0 1m As mentioned in the StatefulSets concept, the Pods in a StatefulSet have a sticky, unique identity. This identity is based on a unique ordinal index that is assigned to each Pod by the StatefulSet controller.The Pods' names take the form <statefulset name>-<ordinal index>. Since the web StatefulSet has two replicas, it creates two Pods, web-0 and web-1.Using stable network identitiesEach Pod has a stable hostname based on its ordinal index. Use kubectl exec to execute the hostname command in each Pod:for i in 0 1; do kubectl exec "web-$i" -- sh -c 'hostname'; done web-0 web-1 Use kubectl run to execute a container that provides the nslookup command from the dnsutils package. Using nslookup on the Pods' hostnames, you can examine their in-cluster DNS addresses:kubectl run -i --tty --image busybox:1.28 dns-test --restart=Never --rm which starts a new shell. In that new shell, run:# Run this in the dns-test container shell nslookup web-0.nginx The output is similar to:Server: 10.0.0.10 Address 1: 10.0.0.10 kube-dns.kube-system.svc.cluster.local Name: web-0.nginx Address 1: 10.244.1.6 nslookup web-1.nginx Server: 10.0.0.10 Address 1: 10.0.0.10 kube-dns.kube-system.svc.cluster.local Name: web-1.nginx Address 1: 10.244.2.6 (and now exit the container shell: exit)The CNAME of the headless service points to SRV records (one for each Pod that is Running and Ready). The SRV records point to A record entries that contain the Pods' IP addresses.In one terminal, watch the StatefulSet's Pods:# Start a new watch # End this watch when you've seen that the delete is finished kubectl get pod --watch -l app=nginx In a second terminal, use kubectl delete to delete all the Pods in the StatefulSet:kubectl delete pod -l app=nginx pod "web-0" deleted pod "web-1" deleted Wait for the StatefulSet to restart them, and for both Pods to transition to Running and Ready:# This should already be running kubectl get pod --watch -l app=nginx NAME READY STATUS RESTARTS AGE web-0 0/1 ContainerCreating 0 0s NAME READY STATUS RESTARTS AGE web-0 1/1 Running 0 2s web-1 0/1 Pending 0 0s web-1 0/1 Pending 0 0s web-1 0/1 ContainerCreating 0 0s web-1 1/1 Running 0 34s Use kubectl exec and kubectl run to view the Pods' hostnames and in-cluster DNS entries. First, view the Pods' hostnames:for i in 0 1; do kubectl exec web-$i -- sh -c 'hostname'; done web-0 web-1 then, run:kubectl run -i --tty --image busybox:1.28 dns-test --restart=Never --rm which starts a new shell.In that new shell, run:# Run this in the dns-test container shell nslookup web-0.nginx The output is similar to:Server: 10.0.0.10 Address 1: 10.0.0.10 kube-dns.kube-system.svc.cluster.local Name: web-0.nginx Address 1: 10.244.1.7 nslookup web-1.nginx Server: 10.0.0.10 Address 1: 10.0.0.10 kube-dns.kube-system.svc.cluster.local Name: web-1.nginx Address 1: 10.244.2.8 (and now exit the container shell: exit)The Pods' ordinals, hostnames, SRV records, and A record names have not changed, but the IP addresses associated with the Pods may have changed. In the cluster used for this tutorial, they have. This is why it is important not to configure other applications to connect to Pods in a StatefulSet by the IP address of a particular Pod (it is OK to connect to Pods by resolving their hostname).Discovery for specific Pods in a StatefulSetIf you need to find and connect to the active members of a StatefulSet, you should query the CNAME of the headless Service (nginx.default.svc.cluster.local). The SRV records associated with the CNAME will contain only the Pods in the StatefulSet that are Running and Ready.If your application already implements connection logic that tests for liveness and readiness, you can use the SRV records of the Pods ( web-0.nginx.default.svc.cluster.local, web-1.nginx.default.svc.cluster.local), as they are stable, and your application will be able to discover the Pods' addresses when they transition to Running and Ready.If your application wants to find any healthy Pod in a StatefulSet, and therefore does not need to track each specific Pod, you could also connect to the IP address of a type: ClusterIP Service, backed by the Pods in that StatefulSet. You can use the same Service that tracks the StatefulSet (specified in the serviceName of the StatefulSet) or a separate Service that selects the right set of Pods.Writing to stable storageGet the PersistentVolumeClaims for web-0 and web-1:kubectl get pvc -l app=nginx The output is similar to:NAME STATUS VOLUME CAPACITY ACCESSMODES AGE www-web-0 Bound pvc-15c268c7-b507-11e6-932f-42010a800002 1Gi RWO 48s www-web-1 Bound pvc-15c79307-b507-11e6-932f-42010a800002 1Gi RWO 48s The StatefulSet controller created two PersistentVolumeClaims that are bound to two PersistentVolumes.As the cluster used in this tutorial is configured to dynamically provision PersistentVolumes, the PersistentVolumes were created and bound automatically.The NGINX webserver, by default, serves an index file from /usr/share/nginx/html/index.html. The volumeMounts field in the StatefulSet's spec ensures that the /usr/share/nginx/html directory is backed by a PersistentVolume.Write the Pods' hostnames to their index.html files and verify that the NGINX webservers serve the hostnames:for i in 0 1; do kubectl exec "web-$i" -- sh -c 'echo "$(hostname)" > /usr/share/nginx/html/index.html'; done for i in 0 1; do kubectl exec -i -t "web-$i" -- curl http://localhost/; done web-0 web-1 Note:If you instead see 403 Forbidden responses for the above curl command, you will need to fix the permissions of the directory mounted by the volumeMounts (due to a bug when using hostPath volumes), by running:for i in 0 1; do kubectl exec web-$i -- chmod 755 /usr/share/nginx/html; donebefore retrying the curl command above.In one terminal, watch the StatefulSet's Pods:# End this watch when you've reached the end of the section. # At the start of "Scaling a StatefulSet" you'll start a new watch. kubectl get pod --watch -l app=nginx In a second terminal, delete all of the StatefulSet's Pods:kubectl delete pod -l app=nginx pod "web-0" deleted pod "web-1" deleted Examine the output of the kubectl get command in the first terminal, and wait for all of the Pods to transition to Running and Ready.# This should already be running kubectl get pod --watch -l app=nginx NAME READY STATUS RESTARTS AGE web-0 0/1 ContainerCreating 0 0s NAME READY STATUS RESTARTS AGE web-0 1/1 Running 0 2s web-1 0/1 Pending 0 0s web-1 0/1 Pending 0 0s web-1 0/1 ContainerCreating 0 0s web-1 1/1 Running 0 34s Verify the web servers continue to serve their hostnames:for i in 0 1; do kubectl exec -i -t "web-$i" -- curl http://localhost/; done web-0 web-1 Even though web-0 and web-1 were rescheduled, they continue to serve their hostnames because the PersistentVolumes associated with their PersistentVolumeClaims are remounted to their volumeMounts. No matter what node web-0and web-1 are scheduled on, their PersistentVolumes will be mounted to the appropriate mount points.Scaling a StatefulSetScaling a StatefulSet refers to increasing or decreasing the number of replicas (horizontal scaling). This is accomplished by updating the replicas field. You can use either kubectl scale or kubectl patch to scale a StatefulSet.Scaling upScaling up means adding more replicas. Provided that your app is able to distribute work across the StatefulSet, the new larger set of Pods can perform more of that work.In one terminal window, watch the Pods in the StatefulSet:# If you already have a watch running, you can continue using that. # Otherwise, start one. # End this watch when there are 5 healthy Pods for the StatefulSet kubectl get pods --watch -l app=nginx In another terminal window, use kubectl scale to scale the number of replicas to 5:kubectl scale sts web --replicas=5 statefulset.apps/web scaled Examine the output of the kubectl get command in the first terminal, and wait for the three additional Pods to transition to Running and Ready.# This should already be running kubectl get pod --watch -l app=nginx NAME READY STATUS RESTARTS AGE web-0 1/1 Running 0 2h web-1 1/1 Running 0 2h NAME READY STATUS RESTARTS AGE web-2 0/1 Pending 0 0s web-2 0/1 Pending 0 0s web-2 0/1 ContainerCreating 0 0s web-2 1/1 Running 0 19s web-3 0/1 Pending 0 0s web-3 0/1 Pending 0 0s web-3 0/1 ContainerCreating 0 0s web-3 1/1 Running 0 18s web-4 0/1 Pending 0 0s web-4 0/1 Pending 0 0s web-4 0/1 ContainerCreating 0 0s web-4 1/1 Running 0 19s The StatefulSet controller scaled the number of replicas. As with StatefulSet creation, the StatefulSet controller created each Pod sequentially with respect to its ordinal index, and it waited for each Pod's predecessor to be Running and Ready before launching the subsequent Pod.Scaling downScaling down means reducing the number of replicas. For example, you might do this because the level of traffic to a service has decreased, and at the current scale there are idle resources.In one terminal, watch the StatefulSet's Pods:# End this watch when there are only 3 Pods for the StatefulSet kubectl get pod --watch -l app=nginx In another terminal, use kubectl patch to scale the StatefulSet back down to three replicas:kubectl patch sts web -p '{"spec":{"replicas":3}}' statefulset.apps/web patched Wait for web-4 and web-3 to transition to Terminating.# This should already be running kubectl get pods --watch -l app=nginx NAME READY STATUS RESTARTS AGE web-0 1/1 Running 0 3h web-1 1/1 Running 0 3h web-2 1/1 Running 0 55s web-3 1/1 Running 0 36s web-4 0/1 ContainerCreating 0 18s NAME READY STATUS RESTARTS AGE web-4 1/1 Running 0 19s web-4 1/1 Terminating 0 24s web-4 1/1 Terminating 0 24s web-3 1/1 Terminating 0 42s web-3 1/1 Terminating 0 42s Ordered Pod terminationThe control plane deleted one Pod at a time, in reverse order with respect to its ordinal index, and it waited for each Pod to be completely shut down before deleting the


# Deployment

DeploymentDeployment enables declarative updates for Pods and ReplicaSets.apiVersion: apps/v1import "k8s.io/api/apps/v1"DeploymentDeployment enables declarative updates for Pods and ReplicaSets.apiVersion: apps/v1kind: Deploymentmetadata (ObjectMeta)Standard object's metadata. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadataspec (DeploymentSpec)Specification of the desired behavior of the Deployment.status (DeploymentStatus)Most recently observed status of the Deployment.DeploymentSpecDeploymentSpec is the specification of the desired behavior of the Deployment.selector (LabelSelector), requiredLabel selector for pods. Existing ReplicaSets whose pods are selected by this will be the ones affected by this deployment. It must match the pod template's labels.template (PodTemplateSpec), requiredTemplate describes the pods that will be created. The only allowed template.spec.restartPolicy value is "Always".replicas (int32)Number of desired pods. This is a pointer to distinguish between explicit zero and not specified. Defaults to 1.minReadySeconds (int32)Minimum number of seconds for which a newly created pod should be ready without any of its container crashing, for it to be considered available. Defaults to 0 (pod will be considered available as soon as it is ready)strategy (DeploymentStrategy)Patch strategy: retainKeysThe deployment strategy to use to replace existing pods with new ones.DeploymentStrategy describes how to replace existing pods with new ones.strategy.type (string)Type of deployment. Can be "Recreate" or "RollingUpdate". Default is RollingUpdate.strategy.rollingUpdate (RollingUpdateDeployment)Rolling update config params. Present only if DeploymentStrategyType = RollingUpdate.Spec to control the desired behavior of rolling update.strategy.rollingUpdate.maxSurge (IntOrString)The maximum number of pods that can be scheduled above the desired number of pods. Value can be an absolute number (ex: 5) or a percentage of desired pods (ex: 10%). This can not be 0 if MaxUnavailable is 0. Absolute number is calculated from percentage by rounding up. Defaults to 25%. Example: when this is set to 30%, the new ReplicaSet can be scaled up immediately when the rolling update starts, such that the total number of old and new pods do not exceed 130% of desired pods. Once old pods have been killed, new ReplicaSet can be scaled up further, ensuring that total number of pods running at any time during the update is at most 130% of desired pods.IntOrString is a type that can hold an int32 or a string. When used in JSON or YAML marshalling and unmarshalling, it produces or consumes the inner type. This allows you to have, for example, a JSON field that can accept a name or number.strategy.rollingUpdate.maxUnavailable (IntOrString)The maximum number of pods that can be unavailable during the update. Value can be an absolute number (ex: 5) or a percentage of desired pods (ex: 10%). Absolute number is calculated from percentage by rounding down. This can not be 0 if MaxSurge is 0. Defaults to 25%. Example: when this is set to 30%, the old ReplicaSet can be scaled down to 70% of desired pods immediately when the rolling update starts. Once new pods are ready, old ReplicaSet can be scaled down further, followed by scaling up the new ReplicaSet, ensuring that the total number of pods available at all times during the update is at least 70% of desired pods.IntOrString is a type that can hold an int32 or a string. When used in JSON or YAML marshalling and unmarshalling, it produces or consumes the inner type. This allows you to have, for example, a JSON field that can accept a name or number.revisionHistoryLimit (int32)The number of old ReplicaSets to retain to allow rollback. This is a pointer to distinguish between explicit zero and not specified. Defaults to 10.progressDeadlineSeconds (int32)The maximum time in seconds for a deployment to make progress before it is considered to be failed. The deployment controller will continue to process failed deployments and a condition with a ProgressDeadlineExceeded reason will be surfaced in the deployment status. Note that progress will not be estimated during the time a deployment is paused. Defaults to 600s.paused (boolean)Indicates that the deployment is paused.DeploymentStatusDeploymentStatus is the most recently observed status of the Deployment.replicas (int32)Total number of non-terminating pods targeted by this deployment (their labels match the selector).availableReplicas (int32)Total number of available non-terminating pods (ready for at least minReadySeconds) targeted by this deployment.readyReplicas (int32)Total number of non-terminating pods targeted by this Deployment with a Ready Condition.unavailableReplicas (int32)Total number of unavailable pods targeted by this deployment. This is the total number of pods that are still required for the deployment to have 100% available capacity. They may either be pods that are running but not yet available or pods that still have not been created.updatedReplicas (int32)Total number of non-terminating pods targeted by this deployment that have the desired template spec.terminatingReplicas (int32)Total number of terminating pods targeted by this deployment. Terminating pods have a non-null .metadata.deletionTimestamp and have not yet reached the Failed or Succeeded .status.phase.This is an alpha field. Enable DeploymentReplicaSetTerminatingReplicas to be able to use this field.collisionCount (int32)Count of hash collisions for the Deployment. The Deployment controller uses this field as a collision avoidance mechanism when it needs to create the name for the newest ReplicaSet.conditions ([]DeploymentCondition)Patch strategy: merge on key typeMap: unique values on key type will be kept during a mergeRepresents the latest available observations of a deployment's current state.DeploymentCondition describes the state of a deployment at a certain point.conditions.status (string), requiredStatus of the condition, one of True, False, Unknown.conditions.type (string), requiredType of deployment condition.conditions.lastTransitionTime (Time)Last time the condition transitioned from one status to another.Time is a wrapper around time.Time which supports correct marshaling to YAML and JSON. Wrappers are provided for many of the factory methods that the time package offers.conditions.lastUpdateTime (Time)The last time this condition was updated.Time is a wrapper around time.Time which supports correct marshaling to YAML and JSON. Wrappers are provided for many of the factory methods that the time package offers.conditions.message (string)A human readable message indicating details about the transition.conditions.reason (string)The reason for the condition's last transition.observedGeneration (int64)The generation observed by the deployment controller.DeploymentListDeploymentList is a list of Deployments.apiVersion: apps/v1kind: DeploymentListmetadata (ListMeta)Standard list metadata.items ([]Deployment), requiredItems is the list of Deployments.Operationsget read the specified DeploymentHTTP RequestGET /apis/apps/v1/namespaces/{namespace}/deployments/{name}Parametersname (in path): string, requiredname of the Deploymentnamespace (in path): string, requirednamespacepretty (in query): stringprettyResponse200 (Deployment): OK401: Unauthorizedget read status of the specified DeploymentHTTP RequestGET /apis/apps/v1/namespaces/{namespace}/deployments/{name}/statusParametersname (in path): string, requiredname of the Deploymentnamespace (in path): string, requirednamespacepretty (in query): stringprettyResponse200 (Deployment): OK401: Unauthorizedlist list or watch objects of kind DeploymentHTTP RequestGET /apis/apps/v1/namespaces/{namespace}/deploymentsParametersnamespace (in path): string, requirednamespaceallowWatchBookmarks (in query): booleanallowWatchBookmarkscontinue (in query): stringcontinuefieldSelector (in query): stringfieldSelectorlabelSelector (in query): stringlabelSelectorlimit (in query): integerlimitpretty (in query): stringprettyresourceVersion (in query): stringresourceVersionresourceVersionMatch (in query): stringresourceVersionMatchsendInitialEvents (in query): booleansendInitialEventstimeoutSeconds (in query): integertimeoutSecondswatch (in query): booleanwatchResponse200 (DeploymentList): OK401: Unauthorizedlist list or watch objects of kind DeploymentHTTP RequestGET /apis/apps/v1/deploymentsParametersallowWatchBookmarks (in query): booleanallowWatchBookmarkscontinue (in query): stringcontinuefieldSelector (in query): stringfieldSelectorlabelSelector (in query): stringlabelSelectorlimit (in query): integerlimitpretty (in query): stringprettyresourceVersion (in query): stringresourceVersionresourceVersionMatch (in query): stringresourceVersionMatchsendInitialEvents (in query): booleansendInitialEventstimeoutSeconds (in query): integertimeoutSecondswatch (in query): booleanwatchResponse200 (DeploymentList): OK401: Unauthorizedcreate create a DeploymentHTTP RequestPOST /apis/apps/v1/namespaces/{namespace}/deploymentsParametersnamespace (in path): string, requirednamespacebody: Deployment, requireddryRun (in query): stringdryRunfieldManager (in query): stringfieldManagerfieldValidation (in query): stringfieldValidationpretty (in query): stringprettyResponse200 (Deployment): OK201 (Deployment): Created202 (Deployment): Accepted401: Unauthorizedupdate replace the specified DeploymentHTTP RequestPUT /apis/apps/v1/namespaces/{namespace}/deployments/{name}Parametersname (in path): string, requiredname of the Deploymentnamespace (in path): string, requirednamespacebody: Deployment, requireddryRun (in query): stringdryRunfieldManager (in query): stringfieldManagerfieldValidation (in query): stringfieldValidationpretty (in query): stringprettyResponse200 (Deployment): OK201 (Deployment): Created401: Unauthorizedupdate replace status of the specified DeploymentHTTP RequestPUT /apis/apps/v1/namespaces/{namespace}/deployments/{name}/statusParametersname (in path): string, requiredname of the Deploymentnamespace (in path): string, requirednamespacebody: Deployment, requireddryRun (in query): stringdryRunfieldManager (in query): stringfieldManagerfieldValidation (in query): stringfieldValidationpretty (in query): stringprettyResponse200 (Deployment): OK201 (Deployment): Created401: Unauthorizedpatch partially update the specified DeploymentHTTP RequestPATCH /apis/apps/v1/namespaces/{namespace}/deployments/{name}Parametersname (in path): string, requiredname of the Deploymentnamespace (in path): string, requirednamespacebody: Patch, requireddryRun (in query): stringdryRunfieldManager (in query): stringfieldManagerfieldValidation (in query): stringfieldValidationforce (in query): booleanforcepretty (in query): stringprettyResponse200 (Deployment): OK201 (Deployment): Created401: Unauthorizedpatch partially update status of the specified DeploymentHTTP RequestPATCH /apis/apps/v1/namespaces/{namespace}/deployments/{name}/statusParametersname (in path): string, requiredname of the Deploymentnamespace (in path): string, requirednamespacebody: Patch, requireddryRun (in query): stringdryRunfieldManager (in query): stringfieldManagerfieldValidation (in query): stringfieldValidationforce (in query): booleanforcepretty (in query): stringprettyResponse200 (Deployment): OK201 (Deployment): Created401: Unauthorizeddelete delete a DeploymentHTTP RequestDELETE /apis/apps/v1/namespaces/{namespace}/deployments/{name}Parametersname (in path): string, requiredname of the Deploymentnamespace (in path): string, requirednamespacebody: DeleteOptionsdryRun (in query): stringdryRungracePeriodSeconds (in query): integergracePeriodSecondsignoreStoreReadErrorWithClusterBreakingPotential (in query): booleanignoreStoreReadErrorWithClusterBreakingPotentialpretty (in query): stringprettypropagationPolicy (in query): stringpropagationPolicyResponse200 (Status): OK202 (Status): Accepted401: Unauthorizeddeletecollection delete collection of DeploymentHTTP RequestDELETE /apis/apps/v1/namespaces/{namespace}/deploymentsParametersnamespace (in path): string, requirednamespacebody: DeleteOptionscontinue (in query): stringcontinuedryRun (in query): stringdryRunfieldSelector (in query): stringfieldSelectorgracePeriodSeconds (in query): integergracePeriodSecondsignoreStoreReadErrorWithClusterBreakingPotential (in query): booleanignoreStoreReadErrorWithClusterBreakingPotentiallabelSelector (in query): stringlabelSelectorlimit (in query): integerlimitpretty (in query): stringprettypropagationPolicy (in query): stringpropagationPolicyresourceVersion (in query): stringresourceVersionresourceVersionMatch (in query): stringresourceVersionMatchsendInitialEvents (in query): booleansendInitialEventstimeoutSeconds (in query): integertimeoutSecondsResponse200 (Status): OK401: UnauthorizedThis page is automatically generated.If you plan to report an issue with this page, mention that the page is auto-generated in your issue description. The fix may need to happen elsewhere in the Kubernetes project.


# Ingress Controllers

Ingress ControllersIn order for an Ingress to work in your cluster, there must be an ingress controller running. You need to select at least one ingress controller and make sure it is set up in your cluster. This page lists common ingress controllers that you can deploy.In order for the Ingress resource to work, the cluster must have an ingress controller running.Unlike other types of controllers which run as part of the kube-controller-manager binary, Ingress controllers are not started automatically with a cluster. Use this page to choose the ingress controller implementation that best fits your cluster.Kubernetes as a project supports and maintains AWS, GCE, and nginx ingress controllers.Additional controllersNote: This section links to third party projects that provide functionality required by Kubernetes. The Kubernetes project authors aren't responsible for these projects, which are listed alphabetically. To add a project to this list, read the content guide before submitting a change. More information.AKS Application Gateway Ingress Controller is an ingress controller that configures the Azure Application Gateway.Alibaba Cloud MSE Ingress is an ingress controller that configures the Alibaba Cloud Native Gateway, which is also the commercial version of Higress.Apache APISIX ingress controller is an Apache APISIX-based ingress controller.Avi Kubernetes Operator provides L4-L7 load-balancing using VMware NSX Advanced Load Balancer.BFE Ingress Controller is a BFE-based ingress controller.Cilium Ingress Controller is an ingress controller powered by Cilium.The Citrix ingress controller works with Citrix Application Delivery Controller.Contour is an Envoy based ingress controller.Emissary-Ingress API Gateway is an Envoy-based ingress controller.EnRoute is an Envoy based API gateway that can run as an ingress controller.Easegress IngressController is an Easegress based API gateway that can run as an ingress controller.F5 BIG-IP Container Ingress Services for Kubernetes lets you use an Ingress to configure F5 BIG-IP virtual servers.FortiADC Ingress Controller support the Kubernetes Ingress resources and allows you to manage FortiADC objects from KubernetesGloo is an open-source ingress controller based on Envoy, which offers API gateway functionality.HAProxy Ingress is an ingress controller for HAProxy.Higress is an Envoy based API gateway that can run as an ingress controller.The HAProxy Ingress Controller for Kubernetes is also an ingress controller for HAProxy.Istio Ingress is an Istio based ingress controller.The Kong Ingress Controller for Kubernetes is an ingress controller driving Kong Gateway.Kusk Gateway is an OpenAPI-driven ingress controller based on Envoy.The NGINX Ingress Controller for Kubernetes works with the NGINX webserver (as a proxy).The ngrok Kubernetes Ingress Controller is an open source controller for adding secure public access to your K8s services using the ngrok platform.The OCI Native Ingress Controller is an Ingress controller for Oracle Cloud Infrastructure which allows you to manage the OCI Load Balancer.OpenNJet Ingress Controller is a OpenNJet-based ingress controller.The Pomerium Ingress Controller is based on Pomerium, which offers context-aware access policy.Skipper HTTP router and reverse proxy for service composition, including use cases like Kubernetes Ingress, designed as a library to build your custom proxy.The Traefik Kubernetes Ingress provider is an ingress controller for the Traefik proxy.Tyk Operator extends Ingress with Custom Resources to bring API Management capabilities to Ingress. Tyk Operator works with the Open Source Tyk Gateway & Tyk Cloud control plane.Voyager is an ingress controller for HAProxy.Wallarm Ingress Controller is an Ingress Controller that provides WAAP (WAF) and API Security capabilities.Using multiple Ingress controllersYou may deploy any number of ingress controllers using ingress class within a cluster. Note the .metadata.name of your ingress class resource. When you create an ingress you would need that name to specify the ingressClassName field on your Ingress object (refer to IngressSpec v1 reference). ingressClassName is a replacement of the older annotation method.If you do not specify an IngressClass for an Ingress, and your cluster has exactly one IngressClass marked as default, then Kubernetes applies the cluster's default IngressClass to the Ingress. You mark an IngressClass as default by setting the ingressclass.kubernetes.io/is-default-class annotation on that IngressClass, with the string value "true".Ideally, all ingress controllers should fulfill this specification, but the various ingress controllers operate slightly differently.Note:Make sure you review your ingress controller's documentation to understand the caveats of choosing it.What's nextLearn more about Ingress.Set up Ingress on Minikube with the NGINX Controller.Items on this page refer to third party products or projects that provide functionality required by Kubernetes. The Kubernetes project authors aren't responsible for those third-party products or projects. See the CNCF website guidelines for more details.You should read the content guide before proposing a change that adds an extra third-party link.


# Node Resource Managers

Node Resource ManagersIn order to support latency-critical and high-throughput workloads, Kubernetes offers a suite of Resource Managers. The managers aim to co-ordinate and optimise the alignment of node's resources for pods configured with a specific requirement for CPUs, devices, and memory (hugepages) resources.Hardware topology alignment policiesTopology Manager is a kubelet component that aims to coordinate the set of components that are responsible for these optimizations. The overall resource management process is governed using the policy you specify. To learn more, read Control Topology Management Policies on a Node.Policies for assigning CPUs to PodsFEATURE STATE: Kubernetes v1.26 [stable] (enabled by default: true)Once a Pod is bound to a Node, the kubelet on that node may need to either multiplex the existing hardware (for example, sharing CPUs across multiple Pods) or allocate hardware by dedicating some resource (for example, assigning one of more CPUs for a Pod's exclusive use).By default, the kubelet uses CFS quota to enforce pod CPU limits. When the node runs many CPU-bound pods, the workload can move to different CPU cores depending on whether the pod is throttled and which CPU cores are available at scheduling time. Many workloads are not sensitive to this migration and thus work fine without any intervention.However, in workloads where CPU cache affinity and scheduling latency significantly affect workload performance, the kubelet allows alternative CPU management policies to determine some placement preferences on the node. This is implemented using the CPU Manager and its policy. There are two available policies:none: the none policy explicitly enables the existing default CPU affinity scheme, providing no affinity beyond what the OS scheduler does automatically. Limits on CPU usage for Guaranteed pods and Burstable pods are enforced using CFS quota.static: the static policy allows containers in Guaranteed pods with integer CPU requests access to exclusive CPUs on the node. This exclusivity is enforced using the cpuset cgroup controller.Note:System services such as the container runtime and the kubelet itself can continue to run on these exclusive CPUs. The exclusivity only extends to other pods.CPU Manager doesn't support offlining and onlining of CPUs at runtime.Static policyThe static policy enables finer-grained CPU management and exclusive CPU assignment. This policy manages a shared pool of CPUs that initially contains all CPUs in the node. The amount of exclusively allocatable CPUs is equal to the total number of CPUs in the node minus any CPU reservations set by the kubelet configuration. CPUs reserved by these options are taken, in integer quantity, from the initial shared pool in ascending order by physical core ID. This shared pool is the set of CPUs on which any containers in BestEffort and Burstable pods run. Containers in Guaranteed pods with fractional CPU requests also run on CPUs in the shared pool. Only containers that are part of a Guaranteed pod and have integer CPU requests are assigned exclusive CPUs.Note:The kubelet requires a CPU reservation greater than zero when the static policy is enabled. This is because a zero CPU reservation would allow the shared pool to become empty.As Guaranteed pods whose containers fit the requirements for being statically assigned are scheduled to the node, CPUs are removed from the shared pool and placed in the cpuset for the container. CFS quota is not used to bound the CPU usage of these containers as their usage is bound by the scheduling domain itself. In others words, the number of CPUs in the container cpuset is equal to the integer CPU limit specified in the pod spec. This static assignment increases CPU affinity and decreases context switches due to throttling for the CPU-bound workload.Consider the containers in the following pod specs:spec: containers: - name: nginx image: nginx The pod above runs in the BestEffort QoS class because no resource requests or limits are specified. It runs in the shared pool.spec: containers: - name: nginx image: nginx resources: limits: memory: "200Mi" requests: memory: "100Mi" The pod above runs in the Burstable QoS class because resource requests do not equal limits and the cpu quantity is not specified. It runs in the shared pool.spec: containers: - name: nginx image: nginx resources: limits: memory: "200Mi" cpu: "2" requests: memory: "100Mi" cpu: "1" The pod above runs in the Burstable QoS class because resource requests do not equal limits. It runs in the shared pool.spec: containers: - name: nginx image: nginx resources: limits: memory: "200Mi" cpu: "2" requests: memory: "200Mi" cpu: "2" The pod above runs in the Guaranteed QoS class because requests are equal to limits. And the container's resource limit for the CPU resource is an integer greater than or equal to one. The nginx container is granted 2 exclusive CPUs.spec: containers: - name: nginx image: nginx resources: limits: memory: "200Mi" cpu: "1.5" requests: memory: "200Mi" cpu: "1.5" The pod above runs in the Guaranteed QoS class because requests are equal to limits. But the container's resource limit for the CPU resource is a fraction. It runs in the shared pool.spec: containers: - name: nginx image: nginx resources: limits: memory: "200Mi" cpu: "2" The pod above runs in the Guaranteed QoS class because only limits are specified and requests are set equal to limits when not explicitly specified. And the container's resource limit for the CPU resource is an integer greater than or equal to one. The nginx container is granted 2 exclusive CPUs.Static policy optionsHere are the available policy options for the static CPU management policy, listed in alphabetical order:align-by-socket (alpha, hidden by default)Align CPUs by physical package / socket boundary, rather than logical NUMA boundaries (available since Kubernetes v1.25)distribute-cpus-across-cores (alpha, hidden by default)Allocate virtual cores, sometimes called hardware threads, across different physical cores (available since Kubernetes v1.31)distribute-cpus-across-numa (beta, visible by default)Spread CPUs across different NUMA domains, aiming for an even balance between the selected domains (available since Kubernetes v1.23)full-pcpus-only (GA, visible by default)Always allocate full physical cores (available since Kubernetes v1.22, GA since Kubernetes v1.33)strict-cpu-reservation (beta, visible by default)Prevent all the pods regardless of their Quality of Service class to run on reserved CPUs (available since Kubernetes v1.32)prefer-align-cpus-by-uncorecache (beta, visible by default)Align CPUs by uncore (Last-Level) cache boundary on a best-effort way (available since Kubernetes v1.32)You can toggle groups of options on and off based upon their maturity level using the following feature gates:CPUManagerPolicyBetaOptions (default enabled). Disable to hide beta-level options.CPUManagerPolicyAlphaOptions (default disabled). Enable to show alpha-level options.You will still have to enable each option using the cpuManagerPolicyOptions field in the kubelet configuration file.For more detail about the individual options you can configure, read on.full-pcpus-onlyIf the full-pcpus-only policy option is specified, the static policy will always allocate full physical cores. By default, without this option, the static policy allocates CPUs using a topology-aware best-fit allocation. On SMT enabled systems, the policy can allocate individual virtual cores, which correspond to hardware threads. This can lead to different containers sharing the same physical cores; this behaviour in turn contributes to the noisy neighbours problem. With the option enabled, the pod will be admitted by the kubelet only if the CPU request of all its containers can be fulfilled by allocating full physical cores. If the pod does not pass the admission, it will be put in Failed state with the message SMTAlignmentError.distribute-cpus-across-numaIf the distribute-cpus-across-numapolicy option is specified, the static policy will evenly distribute CPUs across NUMA nodes in cases where more than one NUMA node is required to satisfy the allocation. By default, the CPUManager will pack CPUs onto one NUMA node until it is filled, with any remaining CPUs simply spilling over to the next NUMA node. This can cause undesired bottlenecks in parallel code relying on barriers (and similar synchronization primitives), as this type of code tends to run only as fast as its slowest worker (which is slowed down by the fact that fewer CPUs are available on at least one NUMA node). By distributing CPUs evenly across NUMA nodes, application developers can more easily ensure that no single worker suffers from NUMA effects more than any other, improving the overall performance of these types of applications.align-by-socketIf the align-by-socket policy option is specified, CPUs will be considered aligned at the socket boundary when deciding how to allocate CPUs to a container. By default, the CPUManager aligns CPU allocations at the NUMA boundary, which could result in performance degradation if CPUs need to be pulled from more than one NUMA node to satisfy the allocation. Although it tries to ensure that all CPUs are allocated from the minimum number of NUMA nodes, there is no guarantee that those NUMA nodes will be on the same socket. By directing the CPUManager to explicitly align CPUs at the socket boundary rather than the NUMA boundary, we are able to avoid such issues. Note, this policy option is not compatible with TopologyManager single-numa-node policy and does not apply to hardware where the number of sockets is greater than number of NUMA nodes.distribute-cpus-across-coresIf the distribute-cpus-across-cores policy option is specified, the static policy will attempt to allocate virtual cores (hardware threads) across different physical cores. By default, the CPUManager tends to pack CPUs onto as few physical cores as possible, which can lead to contention among CPUs on the same physical core and result in performance bottlenecks. By enabling the distribute-cpus-across-cores policy, the static policy ensures that CPUs are distributed across as many physical cores as possible, reducing the contention on the same physical core and thereby improving overall performance. However, it is important to note that this strategy might be less effective when the system is heavily loaded. Under such conditions, the benefit of reducing contention diminishes. Conversely, default behavior can help in reducing inter-core communication overhead, potentially providing better performance under high load conditions.strict-cpu-reservationThe reservedSystemCPUs parameter in KubeletConfiguration, or the deprecated kubelet command line option --reserved-cpus, defines an explicit CPU set for OS system daemons and kubernetes system daemons. More details of this parameter can be found on the Explicitly Reserved CPU List page. By default, this isolation is implemented only for guaranteed pods with integer CPU requests not for burstable and best-effort pods (and guaranteed pods with fractional CPU requests). Admission is only comparing the CPU requests against the allocatable CPUs. Since the CPU limit is higher than the request, the default behaviour allows burstable and best-effort pods to use up the capacity of reservedSystemCPUs and cause host OS services to starve in real life deployments. If the strict-cpu-reservation policy option is enabled, the static policy will not allow any workload to use the CPU cores specified in reservedSystemCPUs.prefer-align-cpus-by-uncorecacheIf the prefer-align-cpus-by-uncorecache policy is specified, the static policy will allocate CPU resources for individual containers such that all CPUs assigned to a container share the same uncore cache block (also known as the Last-Level Cache or LLC). By default, the CPUManager will tightly pack CPU assignments which can result in containers being assigned CPUs from multiple uncore caches. This option enables the CPUManager to allocate CPUs in a way that maximizes the efficient use of the uncore cache. Allocation is performed on a best-effort basis, aiming to affine as many CPUs as possible within the same uncore cache. If the container's CPU requirement exceeds the CPU capacity of a single uncore cache, the CPUManager minimizes the number of uncore caches used in order to maintain optimal uncore cache alignment. Specific workloads can benefit in performance from the reduction of inter-cache latency and noisy neighbors at the cache level. If the CPUManager cannot align optimally while the node has sufficient resources, the container will still be admitted using the default packed behavior.Memory Management PoliciesFEATURE STATE: Kubernetes v1.32 [stable] (enabled by default: true)The Kubernetes Memory Manager enables the feature of guaranteed memory (and hugepages) allocation for pods in the Guaranteed QoS class.The Memory Manager employs hint generation protocol to yield the most suitable NUMA affinity for a pod. The Memory Manager feeds the central manager (Topology Manager) with these affinity hints. Based on both the hints and Topology Manager policy, the pod is rejected or admitted to the node.Moreover, the Memory Manager ensures that the memory which a pod requests is allocated from a minimum number of NUMA nodes.Other resource managersThe configuration of individual managers is elaborated in dedicated documents:Device Manager


# Overview

This is the multi-page printable view of this section. Click here to print.Return to the regular view of this page.OverviewKubernetes is a portable, extensible, open source platform for managing containerized workloads and services, that facilitates both declarative configuration and automation. It has a large, rapidly growing ecosystem. Kubernetes services, support, and tools are widely available.1: Kubernetes Components2: Objects In Kubernetes2.1: Kubernetes Object Management2.2: Object Names and IDs2.3: Labels and Selectors2.4: Namespaces2.5: Annotations2.6: Field Selectors2.7: Finalizers2.8: Owners and Dependents2.9: Recommended Labels3: The Kubernetes APIThis page is an overview of Kubernetes.The name Kubernetes originates from Greek, meaning helmsman or pilot. K8s as an abbreviation results from counting the eight letters between the "K" and the "s". Google open-sourced the Kubernetes project in 2014. Kubernetes combines over 15 years of Google's experience running production workloads at scale with best-of-breed ideas and practices from the community.Why you need Kubernetes and what it can doContainers are a good way to bundle and run your applications. In a production environment, you need to manage the containers that run the applications and ensure that there is no downtime. For example, if a container goes down, another container needs to start. Wouldn't it be easier if this behavior was handled by a system?That's how Kubernetes comes to the rescue! Kubernetes provides you with a framework to run distributed systems resiliently. It takes care of scaling and failover for your application, provides deployment patterns, and more. For example: Kubernetes can easily manage a canary deployment for your system.Kubernetes provides you with:Service discovery and load balancing Kubernetes can expose a container using the DNS name or using their own IP address. If traffic to a container is high, Kubernetes is able to load balance and distribute the network traffic so that the deployment is stable.Storage orchestration Kubernetes allows you to automatically mount a storage system of your choice, such as local storages, public cloud providers, and more.Automated rollouts and rollbacks You can describe the desired state for your deployed containers using Kubernetes, and it can change the actual state to the desired state at a controlled rate. For example, you can automate Kubernetes to create new containers for your deployment, remove existing containers and adopt all their resources to the new container.Automatic bin packing You provide Kubernetes with a cluster of nodes that it can use to run containerized tasks. You tell Kubernetes how much CPU and memory (RAM) each container needs. Kubernetes can fit containers onto your nodes to make the best use of your resources.Self-healing Kubernetes restarts containers that fail, replaces containers, kills containers that don't respond to your user-defined health check, and doesn't advertise them to clients until they are ready to serve.Secret and configuration management Kubernetes lets you store and manage sensitive information, such as passwords, OAuth tokens, and SSH keys. You can deploy and update secrets and application configuration without rebuilding your container images, and without exposing secrets in your stack configuration.Batch execution In addition to services, Kubernetes can manage your batch and CI workloads, replacing containers that fail, if desired.Horizontal scaling Scale your application up and down with a simple command, with a UI, or automatically based on CPU usage.IPv4/IPv6 dual-stack Allocation of IPv4 and IPv6 addresses to Pods and ServicesDesigned for extensibility Add features to your Kubernetes cluster without changing upstream source code.What Kubernetes is notKubernetes is not a traditional, all-inclusive PaaS (Platform as a Service) system. Since Kubernetes operates at the container level rather than at the hardware level, it provides some generally applicable features common to PaaS offerings, such as deployment, scaling, load balancing, and lets users integrate their logging, monitoring, and alerting solutions. However, Kubernetes is not monolithic, and these default solutions are optional and pluggable. Kubernetes provides the building blocks for building developer platforms, but preserves user choice and flexibility where it is important.Kubernetes:Does not limit the types of applications supported. Kubernetes aims to support an extremely diverse variety of workloads, including stateless, stateful, and data-processing workloads. If an application can run in a container, it should run great on Kubernetes.Does not deploy source code and does not build your application. Continuous Integration, Delivery, and Deployment (CI/CD) workflows are determined by organization cultures and preferences as well as technical requirements.Does not provide application-level services, such as middleware (for example, message buses), data-processing frameworks (for example, Spark), databases (for example, MySQL), caches, nor cluster storage systems (for example, Ceph) as built-in services. Such components can run on Kubernetes, and/or can be accessed by applications running on Kubernetes through portable mechanisms, such as the Open Service Broker.Does not dictate logging, monitoring, or alerting solutions. It provides some integrations as proof of concept, and mechanisms to collect and export metrics.Does not provide nor mandate a configuration language/system (for example, Jsonnet). It provides a declarative API that may be targeted by arbitrary forms of declarative specifications.Does not provide nor adopt any comprehensive machine configuration, maintenance, management, or self-healing systems.Additionally, Kubernetes is not a mere orchestration system. In fact, it eliminates the need for orchestration. The technical definition of orchestration is execution of a defined workflow: first do A, then B, then C. In contrast, Kubernetes comprises a set of independent, composable control processes that continuously drive the current state towards the provided desired state. It shouldn't matter how you get from A to C. Centralized control is also not required. This results in a system that is easier to use and more powerful, robust, resilient, and extensible.Historical context for KubernetesLet's take a look at why Kubernetes is so useful by going back in time.Traditional deployment era:Early on, organizations ran applications on physical servers. There was no way to define resource boundaries for applications in a physical server, and this caused resource allocation issues. For example, if multiple applications run on a physical server, there can be instances where one application would take up most of the resources, and as a result, the other applications would underperform. A solution for this would be to run each application on a different physical server. But this did not scale as resources were underutilized, and it was expensive for organizations to maintain many physical servers.Virtualized deployment era:As a solution, virtualization was introduced. It allows you to run multiple Virtual Machines (VMs) on a single physical server's CPU. Virtualization allows applications to be isolated between VMs and provides a level of security as the information of one application cannot be freely accessed by another application.Virtualization allows better utilization of resources in a physical server and allows better scalability because an application can be added or updated easily, reduces hardware costs, and much more. With virtualization you can present a set of physical resources as a cluster of disposable virtual machines.Each VM is a full machine running all the components, including its own operating system, on top of the virtualized hardware.Container deployment era:Containers are similar to VMs, but they have relaxed isolation properties to share the Operating System (OS) among the applications. Therefore, containers are considered lightweight. Similar to a VM, a container has its own filesystem, share of CPU, memory, process space, and more. As they are decoupled from the underlying infrastructure, they are portable across clouds and OS distributions.Containers have become popular because they provide extra benefits, such as:Agile application creation and deployment: increased ease and efficiency of container image creation compared to VM image use.Continuous development, integration, and deployment: provides for reliable and frequent container image build and deployment with quick and efficient rollbacks (due to image immutability).Dev and Ops separation of concerns: create application container images at build/release time rather than deployment time, thereby decoupling applications from infrastructure.Observability: not only surfaces OS-level information and metrics, but also application health and other signals.Environmental consistency across development, testing, and production: runs the same on a laptop as it does in the cloud.Cloud and OS distribution portability: runs on Ubuntu, RHEL, CoreOS, on-premises, on major public clouds, and anywhere else.Application-centric management: raises the level of abstraction from running an OS on virtual hardware to running an application on an OS using logical resources.Loosely coupled, distributed, elastic, liberated micro-services: applications are broken into smaller, independent pieces and can be deployed and managed dynamically – not a monolithic stack running on one big single-purpose machine.Resource isolation: predictable application performance.Resource utilization: high efficiency and density.What's


# Kubernetes API health endpoints

Kubernetes API health endpointsThe Kubernetes API server provides API endpoints to indicate the current status of the API server. This page describes these API endpoints and explains how you can use them.API endpoints for healthThe Kubernetes API server provides 3 API endpoints (healthz, livez and readyz) to indicate the current status of the API server. The healthz endpoint is deprecated (since Kubernetes v1.16), and you should use the more specific livez and readyz endpoints instead. The livez endpoint can be used with the --livez-grace-period flag to specify the startup duration. For a graceful shutdown you can specify the --shutdown-delay-duration flag with the /readyz endpoint. Machines that check the healthz/livez/readyz of the API server should rely on the HTTP status code. A status code 200 indicates the API server is healthy/live/ready, depending on the called endpoint. The more verbose options shown below are intended to be used by human operators to debug their cluster or understand the state of the API server.The following examples will show how you can interact with the health API endpoints.For all endpoints, you can use the verbose parameter to print out the checks and their status. This can be useful for a human operator to debug the current status of the API server, it is not intended to be consumed by a machine:curl -k https://localhost:6443/livez?verbose or from a remote host with authentication:kubectl get --raw='/readyz?verbose' The output will look like this:[+]ping ok [+]log ok [+]etcd ok [+]poststarthook/start-kube-apiserver-admission-initializer ok [+]poststarthook/generic-apiserver-start-informers ok [+]poststarthook/start-apiextensions-informers ok [+]poststarthook/start-apiextensions-controllers ok [+]poststarthook/crd-informer-synced ok [+]poststarthook/bootstrap-controller ok [+]poststarthook/rbac/bootstrap-roles ok [+]poststarthook/scheduling/bootstrap-system-priority-classes ok [+]poststarthook/start-cluster-authentication-info-controller ok [+]poststarthook/start-kube-aggregator-informers ok [+]poststarthook/apiservice-registration-controller ok [+]poststarthook/apiservice-status-available-controller ok [+]poststarthook/kube-apiserver-autoregistration ok [+]autoregister-completion ok [+]poststarthook/apiservice-openapi-controller ok healthz check passed The Kubernetes API server also supports to exclude specific checks. The query parameters can also be combined like in this example:curl -k 'https://localhost:6443/readyz?verbose&exclude=etcd' The output show that the etcd check is excluded:[+]ping ok [+]log ok [+]etcd excluded: ok [+]poststarthook/start-kube-apiserver-admission-initializer ok [+]poststarthook/generic-apiserver-start-informers ok [+]poststarthook/start-apiextensions-informers ok [+]poststarthook/start-apiextensions-controllers ok [+]poststarthook/crd-informer-synced ok [+]poststarthook/bootstrap-controller ok [+]poststarthook/rbac/bootstrap-roles ok [+]poststarthook/scheduling/bootstrap-system-priority-classes ok [+]poststarthook/start-cluster-authentication-info-controller ok [+]poststarthook/start-kube-aggregator-informers ok [+]poststarthook/apiservice-registration-controller ok [+]poststarthook/apiservice-status-available-controller ok [+]poststarthook/kube-apiserver-autoregistration ok [+]autoregister-completion ok [+]poststarthook/apiservice-openapi-controller ok [+]shutdown ok healthz check passed Individual health checksFEATURE STATE: Kubernetes v1.34 [alpha]Each individual health check exposes an HTTP endpoint and can be checked individually. The schema for the individual health checks is /livez/<healthcheck-name> or /readyz/<healthcheck-name>, where livez and readyz can be used to indicate if you want to check the liveness or the readiness of the API server, respectively. The <healthcheck-name> path can be discovered using the verbose flag from above and take the path between [+] and ok. These individual health checks should not be consumed by machines but can be helpful for a human operator to debug a system:curl -k https://localhost:6443/livez/etcd


# Linux Node Swap Behaviors

Linux Node Swap BehaviorsTo allow Kubernetes workloads to use swap, on a Linux node, you must disable the kubelet's default behavior of failing when swap is detected, and specify memory-swap behavior as LimitedSwap:The available choices for swap behavior are:NoSwap(default) Workloads running as Pods on this node do not and cannot use swap. However, processes outside of Kubernetes' scope, such as system daemons (including the kubelet itself!) can utilize swap. This behavior is beneficial for protecting the node from system-level memory spikes, but it does not safeguard the workloads themselves from such spikes.LimitedSwapKubernetes workloads can utilize swap memory. The amount of swap available to a Pod is determined automatically.To learn more, read swap memory management.


# kubectl set subject

kubectl set subjectSynopsisUpdate the user, group, or service account in a role binding or cluster role binding.kubectl set subject (-f FILENAME | TYPE NAME) [--user=username] [--group=groupname] [--serviceaccount=namespace:serviceaccountname] [--dry-run=server|client|none] Examples # Update a cluster role binding for serviceaccount1 kubectl set subject clusterrolebinding admin --serviceaccount=namespace:serviceaccount1 # Update a role binding for user1, user2, and group1 kubectl set subject rolebinding admin --user=user1 --user=user2 --group=group1 # Print the result (in YAML format) of updating rolebinding subjects from a local, without hitting the server kubectl create rolebinding admin --role=admin --user=admin -o yaml --dry-run=client | kubectl set subject --local -f - --user=foo -o yaml Options--allSelect all resources, in the namespace of the specified resource types--allow-missing-template-keys Default: trueIf true, ignore any errors in templates when a field or map key is missing in the template. Only applies to golang and jsonpath output formats.--dry-run string[="unchanged"] Default: "none"Must be "none", "server", or "client". If client strategy, only print the object that would be sent, without sending it. If server strategy, submit server-side request without persisting the resource.--field-manager string Default: "kubectl-set"Name of the manager used to track field ownership.-f, --filename stringsFilename, directory, or URL to files the resource to update the subjects--group stringsGroups to bind to the role-h, --helphelp for subject-k, --kustomize stringProcess the kustomization directory. This flag can't be used together with -f or -R.--localIf true, set subject will NOT contact api-server but run locally.-o, --output stringOutput format. One of: (json, yaml, name, go-template, go-template-file, template, templatefile, jsonpath, jsonpath-as-json, jsonpath-file).-R, --recursiveProcess the directory used in -f, --filename recursively. Useful when you want to manage related manifests organized within the same directory.-l, --selector stringSelector (label query) to filter on, supports '=', '==', '!=', 'in', 'notin'.(e.g. -l key1=value1,key2=value2,key3 in (value3)). Matching objects must satisfy all of the specified label constraints.--serviceaccount stringsService accounts to bind to the role--show-managed-fieldsIf true, keep the managedFields when printing objects in JSON or YAML format.--template stringTemplate string or path to template file to use when -o=go-template, -o=go-template-file. The template format is golang templates [http://golang.org/pkg/text/template/#pkg-overview].--user stringsUsernames to bind to the roleParent Options Inherited--as stringUsername to impersonate for the operation. User could be a regular user or a service account in a namespace.--as-group stringsGroup to impersonate for the operation, this flag can be repeated to specify multiple groups.--as-uid stringUID to impersonate for the operation.--cache-dir string Default: "$HOME/.kube/cache"Default cache directory--certificate-authority stringPath to a cert file for the certificate authority--client-certificate stringPath to a client certificate file for TLS--client-key stringPath to a client key file for TLS--cluster stringThe name of the kubeconfig cluster to use--context stringThe name of the kubeconfig context to use--disable-compressionIf true, opt-out of response compression for all requests to the server--insecure-skip-tls-verifyIf true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure--kubeconfig stringPath to the kubeconfig file to use for CLI requests.--kuberc stringPath to the kuberc file to use for preferences. This can be disabled by exporting KUBECTL_KUBERC=false feature gate or turning off the feature KUBERC=off.--match-server-versionRequire server version to match client version-n, --namespace stringIf present, the namespace scope for this CLI request--password stringPassword for basic authentication to the API server--profile string Default: "none"Name of profile to capture. One of (none|cpu|heap|goroutine|threadcreate|block|mutex)--profile-output string Default: "profile.pprof"Name of the file to write the profile to--request-timeout string Default: "0"The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests.-s, --server stringThe address and port of the Kubernetes API server--storage-driver-buffer-duration duration Default: 1m0sWrites in the storage driver will be buffered for this duration, and committed to the non memory backends as a single transaction--storage-driver-db string Default: "cadvisor"database name--storage-driver-host string Default: "localhost:8086"database host:port--storage-driver-password string Default: "root"database password--storage-driver-secureuse secure connection with database--storage-driver-table string Default: "stats"table name--storage-driver-user string Default: "root"database username--tls-server-name stringServer name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used--token stringBearer token for authentication to the API server--username stringUsername for basic authentication to the API server--version version[=true]--version, --version=raw prints version information and quits; --version=vX.Y.Z... sets the reported version--warnings-as-errorsTreat warnings received from the server as errors and exit with a non-zero exit codeSee Alsokubectl set - Set specific features on objectsThis page is automatically generated.If you plan to report an issue with this page, mention that the page is auto-generated in your issue description. The fix may need to happen elsewhere in the Kubernetes project.