Node AutoscalingAutomatically provision and consolidate the Nodes in your cluster to adapt to demand and optimize cost.In order to run workloads in your cluster, you need Nodes. Nodes in your cluster can be autoscaled - dynamically provisioned, or consolidated to provide needed capacity while optimizing cost. Autoscaling is performed by Node autoscalers.Node provisioningIf there are Pods in a cluster that can't be scheduled on existing Nodes, new Nodes can be automatically added to the cluster—provisioned—to accommodate the Pods. This is especially useful if the number of Pods changes over time, for example as a result of combining horizontal workload with Node autoscaling.Autoscalers provision the Nodes by creating and deleting cloud provider resources backing them. Most commonly, the resources backing the Nodes are Virtual Machines.The main goal of provisioning is to make all Pods schedulable. This goal is not always attainable because of various limitations, including reaching configured provisioning limits, provisioning configuration not being compatible with a particular set of pods, or the lack of cloud provider capacity. While provisioning, Node autoscalers often try to achieve additional goals (for example minimizing the cost of the provisioned Nodes or balancing the number of Nodes between failure domains).There are two main inputs to a Node autoscaler when determining Nodes to provision—Pod scheduling constraints, and Node constraints imposed by autoscaler configuration.Autoscaler configuration may also include other Node provisioning triggers (for example the number of Nodes falling below a configured minimum limit).Note:Provisioning was formerly known as scale-up in Cluster Autoscaler.Pod scheduling constraintsPods can express scheduling constraints to impose limitations on the kind of Nodes they can be scheduled on. Node autoscalers take these constraints into account to ensure that the pending Pods can be scheduled on the provisioned Nodes.The most common kind of scheduling constraints are the resource requests specified by Pod containers. Autoscalers will make sure that the provisioned Nodes have enough resources to satisfy the requests. However, they don't directly take into account the real resource usage of the Pods after they start running. In order to autoscale Nodes based on actual workload resource usage, you can combine horizontal workload autoscaling with Node autoscaling.Other common Pod scheduling constraints include Node affinity, inter-Pod affinity, or a requirement for a particular storage volume.Node constraints imposed by autoscaler configurationThe specifics of the provisioned Nodes (for example the amount of resources, the presence of a given label) depend on autoscaler configuration. Autoscalers can either choose them from a pre-defined set of Node configurations, or use auto-provisioning.Auto-provisioningNode auto-provisioning is a mode of provisioning in which a user doesn't have to fully configure the specifics of the Nodes that can be provisioned. Instead, the autoscaler dynamically chooses the Node configuration based on the pending Pods it's reacting to, as well as pre-configured constraints (for example, the minimum amount of resources or the need for a given label).Node consolidationThe main consideration when running a cluster is ensuring that all schedulable pods are running, whilst keeping the cost of the cluster as low as possible. To achieve this, the Pods' resource requests should utilize as much of the Nodes' resources as possible. From this perspective, the overall Node utilization in a cluster can be used as a proxy for how cost-effective the cluster is.Note:Correctly setting the resource requests of your Pods is as important to the overall cost-effectiveness of a cluster as optimizing Node utilization. Combining Node autoscaling with vertical workload autoscaling can help you achieve this.Nodes in your cluster can be automatically consolidated in order to improve the overall Node utilization, and in turn the cost-effectiveness of the cluster. Consolidation happens through removing a set of underutilized Nodes from the cluster. Optionally, a different set of Nodes can be provisioned to replace them.Consolidation, like provisioning, only considers Pod resource requests and not real resource usage when making decisions.For the purpose of consolidation, a Node is considered empty if it only has DaemonSet and static Pods running on it. Removing empty Nodes during consolidation is more straightforward than non-empty ones, and autoscalers often have optimizations designed specifically for consolidating empty Nodes.Removing non-empty Nodes during consolidation is disruptive—the Pods running on them are terminated, and possibly have to be recreated (for example by a Deployment). However, all such recreated Pods should be able to schedule on existing Nodes in the cluster, or the replacement Nodes provisioned as part of consolidation. No Pods should normally become pending as a result of consolidation.Note:Autoscalers predict how a recreated Pod will likely be scheduled after a Node is provisioned or consolidated, but they don't control the actual scheduling. Because of this, some Pods might become pending as a result of consolidation - if for example a completely new Pod appears while consolidation is being performed.Autoscaler configuration may also enable triggering consolidation by other conditions (for example, the time elapsed since a Node was created), in order to optimize different properties (for example, the maximum lifespan of Nodes in a cluster).The details of how consolidation is performed depend on the configuration of a given autoscaler.Note:Consolidation was formerly known as scale-down in Cluster Autoscaler.AutoscalersThe functionalities described in previous sections are provided by Node autoscalers. In addition to the Kubernetes API, autoscalers also need to interact with cloud provider APIs to provision and consolidate Nodes. This means that they need to be explicitly integrated with each supported cloud provider. The performance and feature set of a given autoscaler can differ between cloud provider integrations.graph TD na[Node autoscaler] k8s[Kubernetes] cp[Cloud Provider] k8s --> |get Pods/Nodes|na na --> |drain Nodes|k8s na --> |create/remove resources backing Nodes|cp cp --> |get resources backing Nodes|na classDef white_on_blue fill:#326ce5,stroke:#fff,stroke-width:4px,color:#fff; classDef blue_on_white fill:#fff,stroke:#bbb,stroke-width:2px,color:#326ce5; class na blue_on_white; class k8s,cp white_on_blue;JavaScript must be enabled to view this contentAutoscaler implementationsCluster Autoscaler and Karpenter are the two Node autoscalers currently sponsored by SIG Autoscaling.From the perspective of a cluster user, both autoscalers should provide a similar Node autoscaling experience. Both will provision new Nodes for unschedulable Pods, and both will consolidate the Nodes that are no longer optimally utilized.Different autoscalers may also provide features outside the Node autoscaling scope described on this page, and those additional features may differ between them.Consult the sections below, and the linked documentation for the individual autoscalers to decide which autoscaler fits your use case better.Cluster AutoscalerCluster Autoscaler adds or removes Nodes to pre-configured Node groups. Node groups generally map to some sort of cloud provider resource group (most commonly a Virtual Machine group). A single instance of Cluster Autoscaler can simultaneously manage multiple Node groups. When provisioning, Cluster Autoscaler will add Nodes to the group that best fits the requests of pending Pods. When consolidating, Cluster Autoscaler always selects specific Nodes to remove, as opposed to just resizing the underlying cloud provider resource group.Additional context:Documentation overviewCloud provider integrationsCluster Autoscaler FAQContactKarpenterKarpenter auto-provisions Nodes based on NodePool configurations provided by the cluster operator. Karpenter handles all aspects of node lifecycle, not just autoscaling. This includes automatically refreshing Nodes once they reach a certain lifetime, and auto-upgrading Nodes when new worker Node images are released. It works directly with individual cloud provider resources (most commonly individual Virtual Machines), and doesn't rely on cloud provider resource groups.Additional context:DocumentationCloud provider integrationsKarpenter FAQContactImplementation comparisonMain differences between Cluster Autoscaler and Karpenter:Cluster Autoscaler provides features related to just Node autoscaling. Karpenter has a wider scope, and also provides features intended for managing Node lifecycle altogether (for example, utilizing disruption to auto-recreate Nodes once they reach a certain lifetime, or auto-upgrade them to new versions).Cluster Autoscaler doesn't support auto-provisioning, the Node groups it can provision from have to be pre-configured. Karpenter supports auto-provisioning, so the user only has to configure a set of constraints for the provisioned Nodes, instead of fully configuring homogenous groups.Cluster Autoscaler provides cloud provider integrations directly, which means that they're a part of the Kubernetes project. For Karpenter, the Kubernetes project publishes Karpenter as a library that cloud providers can integrate with to build a Node autoscaler.Cluster Autoscaler provides integrations with numerous cloud providers, including smaller and less popular providers. There are fewer cloud providers that integrate with Karpenter, including AWS, and Azure.Combine workload and Node autoscalingHorizontal workload autoscalingNode autoscaling usually works in response to Pods—it provisions new Nodes to accommodate unschedulable Pods, and then consolidates the Nodes once they're no longer needed.Horizontal workload autoscaling automatically scales the number of workload replicas to maintain a desired average resource utilization across the replicas. In other words, it automatically creates new Pods in response to application load, and then removes the Pods once the load decreases.You can use Node autoscaling together with horizontal workload autoscaling to autoscale the Nodes in your cluster based on the average real resource utilization of your Pods.If the application load increases, the average utilization of its Pods should also increase, prompting workload autoscaling to create new Pods. Node autoscaling should then provision new Nodes to accommodate the new Pods.Once the application load decreases, workload autoscaling should remove unnecessary Pods. Node autoscaling should, in turn, consolidate the Nodes that are no longer needed.If configured correctly, this pattern ensures that your application always has the Node capacity to handle load spikes if needed, but you don't have to pay for the capacity when it's not needed.Vertical workload autoscalingWhen using Node autoscaling, it's important to set Pod resource requests correctly. If the requests of a given Pod are too low, provisioning a new Node for it might not help the Pod actually run. If the requests of a given Pod are too high, it might incorrectly prevent consolidating its Node.Vertical workload autoscaling automatically adjusts the resource requests of your Pods based on their historical resource usage.You can use Node autoscaling together with vertical workload autoscaling in order to adjust the resource requests of your Pods while preserving Node autoscaling capabilities in your cluster.Caution:When using Node autoscaling, it's not recommended to set up vertical workload autoscaling for DaemonSet Pods. Autoscalers have to predict what DaemonSet Pods on a new Node will look like in order to predict available Node resources. Vertical workload autoscaling might make these predictions unreliable, leading to incorrect scaling decisions.Related componentsThis section describes components providing functionality related to Node autoscaling.DeschedulerThe descheduler is a component providing Node consolidation functionality based on custom policies, as well as other features related to optimizing Nodes and Pods (for example deleting frequently restarting Pods).Workload autoscalers based on cluster sizeCluster Proportional Autoscaler and Cluster Proportional Vertical Autoscaler provide horizontal, and vertical workload autoscaling based on the number of Nodes in the cluster. You can read more in autoscaling based on cluster size.What's nextRead about workload-level autoscaling

Node Autoscaling


Viewing Pods and NodesObjectivesLearn about Kubernetes Pods.Learn about Kubernetes Nodes.Troubleshoot deployed applications.Kubernetes PodsA Pod is a group of one or more application containers (such as Docker) and includes shared storage (volumes), IP address and information about how to run them.When you created a Deployment in Module 2, Kubernetes created a Pod to host your application instance. A Pod is a Kubernetes abstraction that represents a group of one or more application containers (such as Docker), and some shared resources for those containers. Those resources include:Shared storage, as VolumesNetworking, as a unique cluster IP addressInformation about how to run each container, such as the container image version or specific ports to useA Pod models an application-specific "logical host" and can contain different application containers which are relatively tightly coupled. For example, a Pod might include both the container with your Node.js app as well as a different container that feeds the data to be published by the Node.js webserver. The containers in a Pod share an IP Address and port space, are always co-located and co-scheduled, and run in a shared context on the same Node.Pods are the atomic unit on the Kubernetes platform. When we create a Deployment on Kubernetes, that Deployment creates Pods with containers inside them (as opposed to creating containers directly). Each Pod is tied to the Node where it is scheduled, and remains there until termination (according to restart policy) or deletion. In case of a Node failure, identical Pods are scheduled on other available Nodes in the cluster.Pods overviewContainers should only be scheduled together in a single Pod if they are tightly coupled and need to share resources such as disk.NodesA Pod always runs on a Node. A Node is a worker machine in Kubernetes and may be either a virtual or a physical machine, depending on the cluster. Each Node is managed by the control plane. A Node can have multiple pods, and the Kubernetes control plane automatically handles scheduling the pods across the Nodes in the cluster. The control plane's automatic scheduling takes into account the available resources on each Node.Every Kubernetes Node runs at least:Kubelet, a process responsible for communication between the Kubernetes control plane and the Node; it manages the Pods and the containers running on a machine.A container runtime (like Docker) responsible for pulling the container image from a registry, unpacking the container, and running the application.Nodes overviewTroubleshooting with kubectlIn Module 2, you used the kubectl command-line interface. You'll continue to use it in Module 3 to get information about deployed applications and their environments. The most common operations can be done with the following kubectl subcommands:kubectl get - list resourceskubectl describe - show detailed information about a resourcekubectl logs - print the logs from a container in a podkubectl exec - execute a command on a container in a podYou can use these commands to see when applications were deployed, what their current statuses are, where they are running and what their configurations are.Now that we know more about our cluster components and the command line, let's explore our application.Check application configurationLet's verify that the application we deployed in the previous scenario is running. We'll use the kubectl get command and look for existing Pods:kubectl get pods If no pods are running, please wait a couple of seconds and list the Pods again. You can continue once you see one Pod running.Next, to view what containers are inside that Pod and what images are used to build those containers we run the kubectl describe pods command:kubectl describe pods We see here details about the Pod’s container: IP address, the ports used and a list of events related to the lifecycle of the Pod.The output of the describe subcommand is extensive and covers some concepts that we didn’t explain yet, but don’t worry, they will become familiar by the end of this tutorial.Note:The describe subcommand can be used to get detailed information about most of the Kubernetes primitives, including Nodes, Pods, and Deployments. The describe output is designed to be human readable, not to be scripted against.Show the app in the terminalRecall that Pods are running in an isolated, private network - so we need to proxy access to them so we can debug and interact with them. To do this, we'll use the kubectl proxy command to run a proxy in a second terminal. Open a new terminal window, and in that new terminal, run:kubectl proxy Now again, we'll get the Pod name and query that pod directly through the proxy. To get the Pod name and store it in the POD_NAME environment variable:export POD_NAME="$(kubectl get pods -o go-template --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}')" echo Name of the Pod: $POD_NAME To see the output of our application, run a curl request:curl http://localhost:8001/api/v1/namespaces/default/pods/$POD_NAME:8080/proxy/ The URL is the route to the API of the Pod.Note:We don't need to specify the container name, because we only have one container inside the pod.Executing commands on the containerWe can execute commands directly on the container once the Pod is up and running. For this, we use the exec subcommand and use the name of the Pod as a parameter. Let’s list the environment variables:kubectl exec "$POD_NAME" -- env Again, it's worth mentioning that the name of the container itself can be omitted since we only have a single container in the Pod.Next let’s start a bash session in the Pod’s container:kubectl exec -ti $POD_NAME -- bash We have now an open console on the container where we run our NodeJS application. The source code of the app is in the server.js file:cat server.js You can check that the application is up by running a curl command:curl http://localhost:8080 Note:Here we used localhost because we executed the command inside the NodeJS Pod. If you cannot connect to localhost:8080, check to make sure you have run the kubectl exec command and are launching the command from within the Pod.To close your container connection, type exit.What's nextTutorial Using A Service To Expose Your App.Learn more about Pods.Learn more about Nodes.

Viewing Pods and Nodes


MutatingWebhookConfigurationMutatingWebhookConfiguration describes the configuration of and admission webhook that accept or reject and may change the object.apiVersion: admissionregistration.k8s.io/v1import "k8s.io/api/admissionregistration/v1"MutatingWebhookConfigurationMutatingWebhookConfiguration describes the configuration of and admission webhook that accept or reject and may change the object.apiVersion: admissionregistration.k8s.io/v1kind: MutatingWebhookConfigurationmetadata (ObjectMeta)Standard object metadata; More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata.webhooks ([]MutatingWebhook)Patch strategy: merge on key nameMap: unique values on key name will be kept during a mergeWebhooks is a list of webhooks and the affected resources and operations.MutatingWebhook describes an admission webhook and the resources and operations it applies to.webhooks.admissionReviewVersions ([]string), requiredAtomic: will be replaced during a mergeAdmissionReviewVersions is an ordered list of preferred AdmissionReview versions the Webhook expects. API server will try to use first version in the list which it supports. If none of the versions specified in this list supported by API server, validation will fail for this object. If a persisted webhook configuration specifies allowed versions and does not include any versions known to the API Server, calls to the webhook will fail and be subject to the failure policy.webhooks.clientConfig (WebhookClientConfig), requiredClientConfig defines how to communicate with the hook. RequiredWebhookClientConfig contains the information to make a TLS connection with the webhookwebhooks.clientConfig.caBundle ([]byte)caBundle is a PEM encoded CA bundle which will be used to validate the webhook's server certificate. If unspecified, system trust roots on the apiserver are used.webhooks.clientConfig.service (ServiceReference)service is a reference to the service for this webhook. Either service or url must be specified.If the webhook is running within the cluster, then you should use service.ServiceReference holds a reference to Service.legacy.k8s.iowebhooks.clientConfig.service.name (string), requiredname is the name of the service. Requiredwebhooks.clientConfig.service.namespace (string), requirednamespace is the namespace of the service. Requiredwebhooks.clientConfig.service.path (string)path is an optional URL path which will be sent in any request to this service.webhooks.clientConfig.service.port (int32)If specified, the port on the service that hosting webhook. Default to 443 for backward compatibility. port should be a valid port number (1-65535, inclusive).webhooks.clientConfig.url (string)url gives the location of the webhook, in standard URL form (scheme://host:port/path). Exactly one of url or service must be specified.The host should not refer to a service running in the cluster; use the service field instead. The host might be resolved via external DNS in some apiservers (e.g., kube-apiserver cannot resolve in-cluster DNS as that would be a layering violation). host may also be an IP address.Please note that using localhost or 127.0.0.1 as a host is risky unless you take great care to run this webhook on all hosts which run an apiserver which might need to make calls to this webhook. Such installs are likely to be non-portable, i.e., not easy to turn up in a new cluster.The scheme must be "https"; the URL must begin with "https://".A path is optional, and if present may be any string permissible in a URL. You may use the path to pass an arbitrary string to the webhook, for example, a cluster identifier.Attempting to use a user or basic auth e.g. "user:password@" is not allowed. Fragments ("#...") and query parameters ("?...") are not allowed, either.webhooks.name (string), requiredThe name of the admission webhook. Name should be fully qualified, e.g., imagepolicy.kubernetes.io, where "imagepolicy" is the name of the webhook, and kubernetes.io is the name of the organization. Required.webhooks.sideEffects (string), requiredSideEffects states whether this webhook has side effects. Acceptable values are: None, NoneOnDryRun (webhooks created via v1beta1 may also specify Some or Unknown). Webhooks with side effects MUST implement a reconciliation system, since a request may be rejected by a future step in the admission chain and the side effects therefore need to be undone. Requests with the dryRun attribute will be auto-rejected if they match a webhook with sideEffects == Unknown or Some.webhooks.failurePolicy (string)FailurePolicy defines how unrecognized errors from the admission endpoint are handled - allowed values are Ignore or Fail. Defaults to Fail.webhooks.matchConditions ([]MatchCondition)Patch strategy: merge on key nameMap: unique values on key name will be kept during a mergeMatchConditions is a list of conditions that must be met for a request to be sent to this webhook. Match conditions filter requests that have already been matched by the rules, namespaceSelector, and objectSelector. An empty list of matchConditions matches all requests. There are a maximum of 64 match conditions allowed.The exact matching logic is (in order):If ANY matchCondition evaluates to FALSE, the webhook is skipped.If ALL matchConditions evaluate to TRUE, the webhook is called.If any matchCondition evaluates to an error (but none are FALSE):If failurePolicy=Fail, reject the requestIf failurePolicy=Ignore, the error is ignored and the webhook is skippedMatchCondition represents a condition which must by fulfilled for a request to be sent to a webhook.webhooks.matchConditions.expression (string), requiredExpression represents the expression which will be evaluated by CEL. Must evaluate to bool. CEL expressions have access to the contents of the AdmissionRequest and Authorizer, organized into CEL variables:'object' - The object from the incoming request. The value is null for DELETE requests. 'oldObject' - The existing object. The value is null for CREATE requests. 'request' - Attributes of the admission request(/pkg/apis/admission/types.go#AdmissionRequest). 'authorizer' - A CEL Authorizer. May be used to perform authorization checks for the principal (user or service account) of the request. See https://pkg.go.dev/k8s.io/apiserver/pkg/cel/library#Authz 'authorizer.requestResource' - A CEL ResourceCheck constructed from the 'authorizer' and configured with the request resource. Documentation on CEL: https://kubernetes.io/docs/reference/using-api/cel/Required.webhooks.matchConditions.name (string), requiredName is an identifier for this match condition, used for strategic merging of MatchConditions, as well as providing an identifier for logging purposes. A good name should be descriptive of the associated expression. Name must be a qualified name consisting of alphanumeric characters, '-', '' or '.', and must start and end with an alphanumeric character (e.g. 'MyName', or 'my.name', or '123-abc', regex used for validation is '([A-Za-z0-9][-A-Za-z0-9.]*)?[A-Za-z0-9]') with an optional DNS subdomain prefix and '/' (e.g. 'example.com/MyName')Required.webhooks.matchPolicy (string)matchPolicy defines how the "rules" list is used to match incoming requests. Allowed values are "Exact" or "Equivalent".Exact: match a request only if it exactly matches a specified rule. For example, if deployments can be modified via apps/v1, apps/v1beta1, and extensions/v1beta1, but "rules" only included apiGroups:["apps"], apiVersions:["v1"], resources: ["deployments"], a request to apps/v1beta1 or extensions/v1beta1 would not be sent to the webhook.Equivalent: match a request if modifies a resource listed in rules, even via another API group or version. For example, if deployments can be modified via apps/v1, apps/v1beta1, and extensions/v1beta1, and "rules" only included apiGroups:["apps"], apiVersions:["v1"], resources: ["deployments"], a request to apps/v1beta1 or extensions/v1beta1 would be converted to apps/v1 and sent to the webhook.Defaults to "Equivalent"webhooks.namespaceSelector (LabelSelector)NamespaceSelector decides whether to run the webhook on an object based on whether the namespace for that object matches the selector. If the object itself is a namespace, the matching is performed on object.metadata.labels. If the object is another cluster scoped resource, it never skips the webhook.For example, to run the webhook on any objects whose namespace is not associated with "runlevel" of "0" or "1"; you will set the selector as follows: "namespaceSelector": { "matchExpressions": [ { "key": "runlevel", "operator": "NotIn", "values": [ "0", "1" ] } ] }If instead you want to only run the webhook on any objects whose namespace is associated with the "environment" of "prod" or "staging"; you will set the selector as follows: "namespaceSelector": { "matchExpressions": [ { "key": "environment", "operator": "In", "values": [ "prod", "staging" ] } ] }See https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/ for more examples of label selectors.Default to the empty LabelSelector, which matches everything.webhooks.objectSelector (LabelSelector)ObjectSelector decides whether to run the webhook based on if the object has matching labels. objectSelector is evaluated against both the oldObject and newObject that would be sent to the webhook, and is considered to match if either object matches the selector. A null object (oldObject in the case of create, or newObject in the case of delete) or an object that cannot have labels (like a DeploymentRollback or a PodProxyOptions object) is not considered to match. Use the object selector only if the webhook is opt-in, because end users may skip the admission webhook by setting the labels. Default to the empty LabelSelector, which matches everything.webhooks.reinvocationPolicy (string)reinvocationPolicy indicates whether this webhook should be called multiple times as part of a single admission evaluation. Allowed values are "Never" and "IfNeeded".Never: the webhook will not be called more than once in a single admission evaluation.IfNeeded: the webhook will be called at least one additional time as part of the admission evaluation if the object being admitted is modified by other admission plugins after the initial webhook call. Webhooks that specify this option must be idempotent, able to process objects they previously admitted. Note: * the number of additional invocations is not guaranteed to be exactly one. * if additional invocations result in further modifications to the object, webhooks are not guaranteed to be invoked again. * webhooks that use this option may be reordered to minimize the number of additional invocations. * to validate an object after all mutations are guaranteed complete, use a validating admission webhook instead.Defaults to "Never".webhooks.rules ([]RuleWithOperations)Atomic: will be replaced during a mergeRules describes what operations on what resources/subresources the webhook cares about. The webhook cares about an operation if it matches any Rule. However, in order to prevent ValidatingAdmissionWebhooks and MutatingAdmissionWebhooks from putting the cluster in a state which cannot be recovered from without completely disabling the plugin, ValidatingAdmissionWebhooks and MutatingAdmissionWebhooks are never called on admission requests for ValidatingWebhookConfiguration and MutatingWebhookConfiguration objects.RuleWithOperations is a tuple of Operations and Resources. It is recommended to make sure that all the tuple expansions are valid.webhooks.rules.apiGroups ([]string)Atomic: will be replaced during a mergeAPIGroups is the API groups the resources belong to. '' is all groups. If '' is present, the length of the slice must be one. Required.webhooks.rules.apiVersions ([]string)Atomic: will be replaced during a mergeAPIVersions is the API versions the resources belong to. '' is all versions. If '' is present, the length of the slice must be one. Required.webhooks.rules.operations ([]string)Atomic: will be replaced during a mergeOperations is the operations the admission hook cares about - CREATE, UPDATE, DELETE, CONNECT or * for all of those operations and any future admission operations that are added. If '*' is present, the length of the slice must be one. Required.webhooks.rules.resources ([]string)Atomic: will be replaced during a mergeResources is a list of resources this rule applies to.For example: 'pods' means pods. 'pods/log' means the log subresource of pods. '' means all resources, but not subresources. 'pods/' means all subresources of pods. '/scale' means all scale subresources. '/*' means all resources and their subresources.If wildcard is present, the validation rule will ensure resources do not overlap with each other.Depending on the enclosing object, subresources might not be allowed. Required.webhooks.rules.scope (string)scope specifies the scope of this rule. Valid values are "Cluster", "Namespaced", and "" "Cluster" means that only cluster-scoped resources will match this rule. Namespace API objects are cluster-scoped. "Namespaced" means that only namespaced resources will match this rule. "" means that there are no scope restrictions. Subresources match the scope of their parent resource. Default is "*".webhooks.timeoutSeconds (int32)TimeoutSeconds specifies the timeout for this webhook. After the timeout passes, the webhook call will be ignored or the API call will fail based on the failure policy. The timeout value must be between 1 and 30 seconds. Default to 10 seconds.MutatingWebhookConfigurationListMutatingWebhookConfigurationList is a list of MutatingWebhookConfiguration.apiVersion: admissionregistration.k8s.io/v1kind: MutatingWebhookConfigurationListmetadata (ListMeta)Standard list metadata. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kindsitems ([]MutatingWebhookConfiguration), requiredList of MutatingWebhookConfiguration.Operationsget read the specified MutatingWebhookConfigurationHTTP RequestGET /apis/admissionregistration.k8s.io/v1/mutatingwebhookconfigurations/{name}Parametersname (in path): string, requiredname of the MutatingWebhookConfigurationpretty (in query): stringprettyResponse200 (MutatingWebhookConfiguration): OK401: Unauthorizedlist list or watch objects of kind MutatingWebhookConfigurationHTTP RequestGET /apis/admissionregistration.k8s.io/v1/mutatingwebhookconfigurationsParametersallowWatchBookmarks (in query): booleanallowWatchBookmarkscontinue (in query): stringcontinuefieldSelector (in query): stringfieldSelectorlabelSelector (in query): stringlabelSelectorlimit (in query): integerlimitpretty (in query): stringprettyresourceVersion (in query): stringresourceVersionresourceVersionMatch (in query): stringresourceVersionMatchsendInitialEvents (in query): booleansendInitialEventstimeoutSeconds (in query): integertimeoutSecondswatch (in query): booleanwatchResponse200 (MutatingWebhookConfigurationList): OK401: Unauthorizedcreate create a MutatingWebhookConfigurationHTTP RequestPOST /apis/admissionregistration.k8s.io/v1/mutatingwebhookconfigurationsParametersbody: MutatingWebhookConfiguration, requireddryRun (in query): stringdryRunfieldManager (in query): stringfieldManagerfieldValidation (in query): stringfieldValidationpretty (in query): stringprettyResponse200 (MutatingWebhookConfiguration): OK201 (MutatingWebhookConfiguration): Created202 (MutatingWebhookConfiguration): Accepted401: Unauthorizedupdate replace the specified MutatingWebhookConfigurationHTTP RequestPUT /apis/admissionregistration.k8s.io/v1/mutatingwebhookconfigurations/{name}Parametersname (in path): string, requiredname of the MutatingWebhookConfigurationbody: MutatingWebhookConfiguration, requireddryRun (in query): stringdryRunfieldManager (in query): stringfieldManagerfieldValidation (in query): stringfieldValidationpretty (in query): stringprettyResponse200 (MutatingWebhookConfiguration): OK201 (MutatingWebhookConfiguration): Created401: Unauthorizedpatch partially update the specified MutatingWebhookConfigurationHTTP RequestPATCH /apis/admissionregistration.k8s.io/v1/mutatingwebhookconfigurations/{name}Parametersname (in path): string, requiredname of the MutatingWebhookConfigurationbody: Patch, requireddryRun (in query): stringdryRunfieldManager (in query): stringfieldManagerfieldValidation (in query): stringfieldValidationforce (in query): booleanforcepretty (in query): stringprettyResponse200 (MutatingWebhookConfiguration): OK201 (MutatingWebhookConfiguration): Created401: Unauthorizeddelete delete a MutatingWebhookConfigurationHTTP RequestDELETE /apis/admissionregistration.k8s.io/v1/mutatingwebhookconfigurations/{name}Parametersname (in path): string, requiredname of the MutatingWebhookConfigurationbody: DeleteOptionsdryRun (in query): stringdryRungracePeriodSeconds (in query): integergracePeriodSecondsignoreStoreReadErrorWithClusterBreakingPotential (in query): booleanignoreStoreReadErrorWithClusterBreakingPotentialpretty (in query): stringprettypropagationPolicy (in query): stringpropagationPolicyResponse200 (Status): OK202 (Status): Accepted401: Unauthorizeddeletecollection delete collection of MutatingWebhookConfigurationHTTP RequestDELETE /apis/admissionregistration.k8s.io/v1/mutatingwebhookconfigurationsParametersbody: DeleteOptionscontinue (in query): stringcontinuedryRun (in query): stringdryRunfieldSelector (in query): stringfieldSelectorgracePeriodSeconds (in query): integergracePeriodSecondsignoreStoreReadErrorWithClusterBreakingPotential (in query): booleanignoreStoreReadErrorWithClusterBreakingPotentiallabelSelector (in query): stringlabelSelectorlimit (in query): integerlimitpretty (in query): stringprettypropagationPolicy (in query): stringpropagationPolicyresourceVersion (in query): stringresourceVersionresourceVersionMatch (in query): stringresourceVersionMatchsendInitialEvents (in query): booleansendInitialEventstimeoutSeconds (in query): integertimeoutSecondsResponse200 (Status): OK401: UnauthorizedThis page is automatically generated.If you plan to report an issue with this page, mention that the page is auto-generated in your issue description. The fix may need to happen elsewhere in the Kubernetes project.

MutatingWebhookConfiguration


kubectlSynopsiskubectl controls the Kubernetes cluster manager.Find more information at: https://kubernetes.io/docs/reference/kubectl/kubectl [flags] Options--as stringUsername to impersonate for the operation. User could be a regular user or a service account in a namespace.--as-group stringsGroup to impersonate for the operation, this flag can be repeated to specify multiple groups.--as-uid stringUID to impersonate for the operation.--cache-dir string Default: "$HOME/.kube/cache"Default cache directory--certificate-authority stringPath to a cert file for the certificate authority--client-certificate stringPath to a client certificate file for TLS--client-key stringPath to a client key file for TLS--cluster stringThe name of the kubeconfig cluster to use--context stringThe name of the kubeconfig context to use--disable-compressionIf true, opt-out of response compression for all requests to the server-h, --helphelp for kubectl--insecure-skip-tls-verifyIf true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure--kubeconfig stringPath to the kubeconfig file to use for CLI requests.--kuberc stringPath to the kuberc file to use for preferences. This can be disabled by exporting KUBECTL_KUBERC=false feature gate or turning off the feature KUBERC=off.--match-server-versionRequire server version to match client version-n, --namespace stringIf present, the namespace scope for this CLI request--password stringPassword for basic authentication to the API server--profile string Default: "none"Name of profile to capture. One of (none|cpu|heap|goroutine|threadcreate|block|mutex)--profile-output string Default: "profile.pprof"Name of the file to write the profile to--request-timeout string Default: "0"The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests.-s, --server stringThe address and port of the Kubernetes API server--storage-driver-buffer-duration duration Default: 1m0sWrites in the storage driver will be buffered for this duration, and committed to the non memory backends as a single transaction--storage-driver-db string Default: "cadvisor"database name--storage-driver-host string Default: "localhost:8086"database host:port--storage-driver-password string Default: "root"database password--storage-driver-secureuse secure connection with database--storage-driver-table string Default: "stats"table name--storage-driver-user string Default: "root"database username--tls-server-name stringServer name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used--token stringBearer token for authentication to the API server--user stringThe name of the kubeconfig user to use--username stringUsername for basic authentication to the API server--version version[=true]--version, --version=raw prints version information and quits; --version=vX.Y.Z... sets the reported version--warnings-as-errorsTreat warnings received from the server as errors and exit with a non-zero exit codeSee Alsokubectl annotate - Update the annotations on a resourcekubectl api-resources - Print the supported API resources on the serverkubectl api-versions - Print the supported API versions on the server, in the form of "group/version"kubectl apply - Apply a configuration to a resource by file name or stdinkubectl attach - Attach to a running containerkubectl auth - Inspect authorizationkubectl autoscale - Auto-scale a deployment, replica set, stateful set, or replication controllerkubectl certificate - Modify certificate resourceskubectl cluster-info - Display cluster informationkubectl completion - Output shell completion code for the specified shell (bash, zsh, fish, or powershell)kubectl config - Modify kubeconfig fileskubectl cordon - Mark node as unschedulablekubectl cp - Copy files and directories to and from containerskubectl create - Create a resource from a file or from stdinkubectl debug - Create debugging sessions for troubleshooting workloads and nodeskubectl delete - Delete resources by file names, stdin, resources and names, or by resources and label selectorkubectl describe - Show details of a specific resource or group of resourceskubectl diff - Diff the live version against a would-be applied versionkubectl drain - Drain node in preparation for maintenancekubectl edit - Edit a resource on the serverkubectl events - List eventskubectl exec - Execute a command in a containerkubectl explain - Get documentation for a resourcekubectl expose - Take a replication controller, service, deployment or pod and expose it as a new Kubernetes servicekubectl get - Display one or many resourceskubectl kustomize - Build a kustomization target from a directory or URLkubectl label - Update the labels on a resourcekubectl logs - Print the logs for a container in a podkubectl options - Print the list of flags inherited by all commandskubectl patch - Update fields of a resourcekubectl plugin - Provides utilities for interacting with pluginskubectl port-forward - Forward one or more local ports to a podkubectl proxy - Run a proxy to the Kubernetes API serverkubectl replace - Replace a resource by file name or stdinkubectl rollout - Manage the rollout of a resourcekubectl run - Run a particular image on the clusterkubectl scale - Set a new size for a deployment, replica set, or replication controllerkubectl set - Set specific features on objectskubectl taint - Update the taints on one or more nodeskubectl top - Display resource (CPU/memory) usagekubectl uncordon - Mark node as schedulablekubectl version - Print the client and server version informationkubectl wait - Experimental: Wait for a specific condition on one or many resourcesThis page is automatically generated.If you plan to report an issue with this page, mention that the page is auto-generated in your issue description. The fix may need to happen elsewhere in the Kubernetes project.

kubectl


This is the multi-page printable view of this section. Click here to print.Return to the regular view of this page.Cluster AdministrationLower-level detail relevant to creating or administering a Kubernetes cluster.1: Node Shutdowns2: Swap memory management3: Node Autoscaling4: Certificates5: Cluster Networking6: Admission Webhook Good Practices7: Good practices for Dynamic Resource Allocation as a Cluster Admin8: Logging Architecture9: Compatibility Version For Kubernetes Control Plane Components10: Metrics For Kubernetes System Components11: Metrics for Kubernetes Object States12: System Logs13: Traces For Kubernetes System Components14: Proxies in Kubernetes15: API Priority and Fairness16: Installing Addons17: Coordinated Leader ElectionThe cluster administration overview is for anyone creating or administering a Kubernetes cluster. It assumes some familiarity with core Kubernetes concepts.Planning a clusterSee the guides in Setup for examples of how to plan, set up, and configure Kubernetes clusters. The solutions listed in this article are called distros.Note:Not all distros are actively maintained. Choose distros which have been tested with a recent version of Kubernetes.Before choosing a guide, here are some considerations:Do you want to try out Kubernetes on your computer, or do you want to build a high-availability, multi-node cluster? Choose distros best suited for your needs.Will you be using a hosted Kubernetes cluster, such as Google Kubernetes Engine, or hosting your own cluster?Will your cluster be on-premises, or in the cloud (IaaS)? Kubernetes does not directly support hybrid clusters. Instead, you can set up multiple clusters.If you are configuring Kubernetes on-premises, consider which networking model fits best.Will you be running Kubernetes on "bare metal" hardware or on virtual machines (VMs)?Do you want to run a cluster, or do you expect to do active development of Kubernetes project code? If the latter, choose an actively-developed distro. Some distros only use binary releases, but offer a greater variety of choices.Familiarize yourself with the components needed to run a cluster.Managing a clusterLearn how to manage nodes.Read about Node autoscaling.Learn how to set up and manage the resource quota for shared clusters.Securing a clusterGenerate Certificates describes the steps to generate certificates using different tool chains.Kubernetes Container Environment describes the environment for Kubelet managed containers on a Kubernetes node.Controlling Access to the Kubernetes API describes how Kubernetes implements access control for its own API.Authenticating explains authentication in Kubernetes, including the various authentication options.Authorization is separate from authentication, and controls how HTTP calls are handled.Using Admission Controllers explains plug-ins which intercepts requests to the Kubernetes API server after authentication and authorization.Admission Webhook Good Practices provides good practices and considerations when designing mutating admission webhooks and validating admission webhooks.Using Sysctls in a Kubernetes Cluster describes to an administrator how to use the sysctl command-line tool to set kernel parameters .Auditing describes how to interact with Kubernetes' audit logs.Securing the kubeletControl Plane-Node communicationTLS bootstrappingKubelet authentication/authorizationOptional Cluster ServicesDNS Integration describes how to resolve a DNS name directly to a Kubernetes service.Logging and Monitoring Cluster Activity explains how logging in Kubernetes works and how to implement it.1 - Node ShutdownsIn a Kubernetes cluster, a node can be shut down in a planned graceful way or unexpectedly because of reasons such as a power outage or something else external. A node shutdown could lead to workload failure if the node is not drained before the shutdown. A node shutdown can be either graceful or non-graceful.Graceful node shutdownThe kubelet attempts to detect node system shutdown and terminates pods running on the node.Kubelet ensures that pods follow the normal pod termination process during the node shutdown. During node shutdown, the kubelet does not accept new Pods (even if those Pods are already bound to the node).Enabling graceful node shutdownFEATURE STATE: Kubernetes v1.21 [beta] (enabled by default: true)On Linux, the graceful node shutdown feature is controlled with the GracefulNodeShutdown feature gate which is enabled by default in 1.21.Note:The graceful node shutdown feature depends on systemd since it takes advantage of systemd inhibitor locks to delay the node shutdown with a given duration.FEATURE STATE: Kubernetes v1.34 [beta] (enabled by default: true)On Windows, the graceful node shutdown feature is controlled with the WindowsGracefulNodeShutdown feature gate which is introduced in 1.32 as an alpha feature. In Kubernetes 1.34 the feature is Beta and is enabled by default.Note:The Windows graceful node shutdown feature depends on kubelet running as a Windows service, it will then have a registered service control handler to delay the preshutdown event with a given duration.Windows graceful node shutdown can not be cancelled.If kubelet is not running as a Windows service, it will not be able to set and monitor the Preshutdown event, the node will have to go through the Non-Graceful Node Shutdown procedure mentioned above.In the case where the Windows graceful node shutdown feature is enabled, but the kubelet is not running as a Windows service, the kubelet will continue running instead of failing. However, it will log an error indicating that it needs to be run as a Windows service.Configuring graceful node shutdownNote that by default, both configuration options described below, shutdownGracePeriod and shutdownGracePeriodCriticalPods, are set to zero, thus not activating the graceful node shutdown functionality. To activate the feature, both options should be configured appropriately and set to non-zero values.Once the kubelet is notified of a node shutdown, it sets a NotReady condition on the Node, with the reason set to "node is shutting down". The kube-scheduler honors this condition and does not schedule any Pods onto the affected node; other third-party schedulers are expected to follow the same logic. This means that new Pods won't be scheduled onto that node and therefore none will start.The kubelet also rejects Pods during the PodAdmission phase if an ongoing node shutdown has been detected, so that even Pods with a toleration for node.kubernetes.io/not-ready:NoSchedule do not start there.When kubelet is setting that condition on its Node via the API, the kubelet also begins terminating any Pods that are running locally.During a graceful shutdown, kubelet terminates pods in two phases:Terminate regular pods running on the node.Terminate critical pods running on the node.The graceful node shutdown feature is configured with two KubeletConfiguration options:shutdownGracePeriod:Specifies the total duration that the node should delay the shutdown by. This is the total grace period for pod termination for both regular and critical pods.shutdownGracePeriodCriticalPods:Specifies the duration used to terminate critical pods during a node shutdown. This value should be less than shutdownGracePeriod.Note:There are cases when Node termination was cancelled by the system (or perhaps manually by an administrator). In either of those situations the Node will return to the Ready state. However, Pods which already started the process of termination will not be restored by kubelet and will need to be re-scheduled.For example, if shutdownGracePeriod=30s, and shutdownGracePeriodCriticalPods=10s, kubelet will delay the node shutdown by 30 seconds. During the shutdown, the first 20 (30-10) seconds would be reserved for gracefully terminating normal pods, and the last 10 seconds would be reserved for terminating critical pods.Note:When pods were evicted during the graceful node shutdown, they are marked as shutdown. Running kubectl get pods shows the status of the evicted pods as Terminated. And kubectl describe pod indicates that the pod was evicted because of node shutdown:Reason: Terminated Message: Pod was terminated in response to imminent node shutdown. Pod Priority based graceful node shutdownFEATURE STATE: Kubernetes v1.24 [beta] (enabled by default: true)To provide more flexibility during graceful node shutdown around the ordering of pods during shutdown, graceful node shutdown honors the PriorityClass for Pods, provided that you enabled this feature in your cluster. The feature allows cluster administrators to explicitly define the ordering of pods during graceful node shutdown based on priority classes.The Graceful Node Shutdown feature, as described above, shuts down pods in two phases, non-critical pods, followed by critical pods. If additional flexibility is needed to explicitly define the ordering of pods during shutdown in a more granular way, pod priority based graceful shutdown can be used.When graceful node shutdown honors pod priorities, this makes it possible to do graceful node shutdown in multiple phases, each phase shutting down a particular priority class of pods. The kubelet can be configured with the exact phases and shutdown time per phase.Assuming the following custom pod priority classes in a cluster,Pod priority class namePod priority class valuecustom-class-a100000custom-class-b10000custom-class-c1000regular/unset0Within the kubelet configuration the settings for shutdownGracePeriodByPodPriority could look like:Pod priority class valueShutdown period10000010 seconds10000180 seconds1000120 seconds060 secondsThe corresponding kubelet config YAML configuration would be:shutdownGracePeriodByPodPriority: - priority: 100000 shutdownGracePeriodSeconds: 10 - priority: 10000 shutdownGracePeriodSeconds: 180 - priority: 1000 shutdownGracePeriodSeconds: 120 - priority: 0 shutdownGracePeriodSeconds: 60 The above table implies that any pod with priority value >= 100000 will get just 10 seconds to shut down, any pod with value >= 10000 and < 100000 will get 180 seconds to shut down, any pod with value >= 1000 and < 10000 will get 120 seconds to shut down. Finally, all other pods will get 60 seconds to shut down.One doesn't have to specify values corresponding to all of the classes. For example, you could instead use these settings:Pod priority class valueShutdown period100000300 seconds1000120 seconds060 secondsIn the above case, the pods with custom-class-b will go into the same bucket as custom-class-c for shutdown.If there are no pods in a particular range, then the kubelet does not wait for pods in that priority range. Instead, the kubelet immediately skips to the next priority class value range.If this feature is enabled and no configuration is provided, then no ordering action will be taken.Using this feature requires enabling the GracefulNodeShutdownBasedOnPodPriority feature gate, and setting ShutdownGracePeriodByPodPriority in the kubelet config to the desired configuration containing the pod priority class values and their respective shutdown periods.Note:The ability to take Pod priority into account during graceful node shutdown was introduced as an Alpha feature in Kubernetes v1.23. In Kubernetes 1.34 the feature is Beta and is enabled by default.Metrics graceful_shutdown_start_time_seconds and graceful_shutdown_end_time_seconds are emitted under the kubelet subsystem to monitor node shutdowns.Non-graceful node shutdown handlingFEATURE STATE: Kubernetes v1.28 [stable] (enabled by default: true)A node shutdown action may not be detected by kubelet's Node Shutdown Manager, either because the command does not trigger the inhibitor locks mechanism used by kubelet or because of a user error, i.e., the ShutdownGracePeriod and ShutdownGracePeriodCriticalPods are not configured properly. Please refer to above section Graceful Node Shutdown for more details.When a node is shutdown but not detected by kubelet's Node Shutdown Manager, the pods that are part of a StatefulSet will be stuck in terminating status on the shutdown node and cannot move to a new running node. This is because kubelet on the shutdown node is not available to delete the pods so the StatefulSet cannot create a new pod with the same name. If there are volumes used by the pods, the VolumeAttachments will not be deleted from the original shutdown node so the volumes used by these pods cannot be attached to a new running node. As a result, the application running on the StatefulSet cannot function properly. If the original shutdown node comes up, the pods will be deleted by kubelet and new pods will be created on a different running node. If the original shutdown node does not come up, these pods will be stuck in terminating status on the shutdown node forever.To mitigate the above situation, a user can manually add the taint node.kubernetes.io/out-of-service with either NoExecute or NoSchedule effect to a Node marking it out-of-service. If a Node is marked out-of-service with this taint, the pods on the node will be forcefully deleted if there are no matching tolerations on it and volume detach operations for the pods terminating on the node will happen immediately. This allows the Pods on the out-of-service node to recover quickly on a different node.During a non-graceful shutdown, Pods are terminated in the two phases:Force delete the Pods that do not have matching out-of-service tolerations.Immediately perform detach volume operation for such pods.Note:Before adding the taint node.kubernetes.io/out-of-service, it should be verified that the node is already in shutdown or power off state (not in the middle of restarting).The user is required to manually remove the out-of-service taint after the pods are moved to a new node and the user has checked that the shutdown node has been recovered since the user was the one who originally added the taint.Forced storage detach on timeoutIn any situation where a pod deletion has not succeeded for 6 minutes, kubernetes will force detach volumes being unmounted if the node is unhealthy at that instant. Any workload still running on the node that uses a force-detached volume will cause a violation of the CSI specification, which states that ControllerUnpublishVolume "must be called after all NodeUnstageVolume and NodeUnpublishVolume on the volume are called and succeed". In such circumstances, volumes on the node in question might encounter data corruption.The forced storage detach behaviour is optional; users might opt to use the "Non-graceful node shutdown" feature instead.Force storage detach on timeout can be disabled by setting the disable-force-detach-on-timeout config field in kube-controller-manager. Disabling the force detach on timeout feature means that a volume that is hosted on a node that is unhealthy for more than 6 minutes will not have its associated VolumeAttachment deleted.After this setting has been applied, unhealthy pods still attached to volumes must be recovered via the Non-Graceful Node Shutdown procedure mentioned above.Note:Caution must be taken while using the Non-Graceful Node Shutdown procedure.Deviation from the steps documented above can result in data corruption.What's nextLearn more about the following:Blog: Non-Graceful Node Shutdown.Cluster Architecture: Nodes.2 - Swap memory managementKubernetes can be configured to use swap memory on a node, allowing the kernel to free up physical memory by swapping out pages to backing storage. This is useful for multiple use-cases. For example, nodes running workloads that can benefit from using swap, such as those that have large memory footprints but only access a portion of that memory at any given time. It also helps prevent Pods from being terminated during memory pressure spikes, shields nodes from system-level memory spikes that might compromise its stability, allows for more flexible memory management on the node, and much more.To learn about configuring swap in your cluster, read Configuring swap memory on Kubernetes nodes.Operating system supportLinux nodes support swap; you need to configure each node to enable it. By default, the kubelet will not start on a Linux node that has swap enabled.Windows nodes require swap space. By default, the kubelet does not start on a Windows node that has swap disabled.How does it work?There are a number of possible ways that one could envision swap use on a node. If kubelet is already running on a node, it would need to be restarted after swap is provisioned in order to identify it.When kubelet starts on a node in which swap is provisioned and available (with the failSwapOn: false configuration), kubelet will:Be able to start on this swap-enabled node.Direct the Container Runtime Interface (CRI) implementation, often referred to as the container runtime, to allocate zero swap memory to Kubernetes workloads by default.Swap configuration on a node is exposed to a cluster admin via the memorySwap in the KubeletConfiguration. As a cluster administrator, you can specify the node's behaviour in the presence of swap memory by setting memorySwap.swapBehavior.Swap behaviorsYou need to pick a swap behavior to use. Different nodes in your cluster can use different swap behaviors.The swap behaviors you can choose for Linux nodes are:NoSwap (default)Workloads running as Pods on this node do not and cannot use swap.LimitedSwapKubernetes workloads can utilize swap memory.Note:If you choose the NoSwap behavior, and you configure the kubelet to tolerate swap space (failSwapOn: false), then your workloads don't use any swap.However, processes outside of Kubernetes-managed containers, such as systemi services (and even the kubelet itself!) can utilize swap.You can read configuring swap memory on Kubernetes nodes to learn about enabling swap for your cluster.Container runtime integrationThe kubelet uses the container runtime API, and directs the container runtime to apply specific configuration (for example, in the cgroup v2 case, memory.swap.max) in a manner that will enable the desired swap configuration for a container. For runtimes that use control groups, or cgroups, the container runtime is then responsible for writing these settings to the container-level cgroup.Observability for swap useNode and container level metric statisticsKubelet now collects node and container level metric statistics, which can be accessed at the /metrics/resource (which is used mainly by monitoring tools like Prometheus) and /stats/summary (which is used mainly by Autoscalers) kubelet HTTP endpoints. This allows clients who can directly request the kubelet to monitor swap usage and remaining swap memory when using LimitedSwap. Additionally, a machine_swap_bytes metric has been added to cadvisor to show the total physical swap capacity of the machine. See this page for more info.For example, these /metrics/resource are supported:node_swap_usage_bytes: Current swap usage of the node in bytes.container_swap_usage_bytes: Current amount of the container swap usage in bytes.container_swap_limit_bytes: Current amount of the container swap limit in bytes.Using kubectl top --show-swapQuerying metrics is valuable, but somewhat cumbersome, as these metrics are designed to be used by software rather than humans. In order to consume this data in a more user-friendly way, the kubectl top command has been extended to support swap metrics, using the --show-swap flag.In order to receive information about swap usage on nodes, kubectl top nodes --show-swap can be used:kubectl top nodes --show-swap This will result in an output similar to:NAME CPU(cores) CPU(%) MEMORY(bytes) MEMORY(%) SWAP(bytes) SWAP(%) node1 1m 10% 2Mi 10% 1Mi 0% node2 5m 10% 6Mi 10% 2Mi 0% node3 3m 10% 4Mi 10% <unknown> <unknown> In order to receive information about swap usage by pods, kubectl top nodes --show-swap can be used:kubectl top pod -n kube-system --show-swap This will result in an output similar to:NAME CPU(cores) MEMORY(bytes) SWAP(bytes) coredns-58d5bc5cdb-5nbk4 2m 19Mi 0Mi coredns-58d5bc5cdb-jsh26 3m 37Mi 0Mi etcd-node01 51m 143Mi 5Mi kube-apiserver-node01 98m 824Mi 16Mi kube-controller-manager-node01 20m 135Mi 9Mi kube-proxy-ffgs2 1m 24Mi 0Mi kube-proxy-fhvwx 1m 39Mi 0Mi kube-scheduler-node01 13m 69Mi 0Mi metrics-server-8598789fdb-d2kcj 5m 26Mi 0Mi Nodes to report swap capacity as part of node statusA new node status field is now added, node.status.nodeInfo.swap.capacity, to report the swap capacity of a node.As an example, the following command can be used to retrieve the swap capacity of the nodes in a cluster:kubectl get nodes -o go-template='{{range .items}}{{.metadata.name}}: {{if .status.nodeInfo.swap.capacity}}{{.status.nodeInfo.swap.capacity}}{{else}}<unknown>{{end}}{{"\n"}}{{end}}' This will result in an output similar to:node1: 21474836480 node2: 42949664768 node3: <unknown> Note:The <unknown> value indicates that the .status.nodeInfo.swap.capacity field is not set for that Node. This probably means that the node does not have swap provisioned, or less likely, that the kubelet is not able to determine the swap capacity of the node.Swap discovery using Node Feature Discovery (NFD)Node Feature Discovery is a Kubernetes addon for detecting hardware features and configuration. It can be utilized to discover which nodes are provisioned with swap.As an example, to figure out which nodes are provisioned with swap, use the following command:kubectl get nodes -o jsonpath='{range .items[?(@.metadata.labels.feature\.node\.kubernetes\.io/memory-swap)]}{.metadata.name}{"\t"}{.metadata.labels.feature\.node\.kubernetes\.io/memory-swap}{"\n"}{end}' This will result in an output similar to:k8s-worker1: true k8s-worker2: true k8s-worker3: false In this example, swap is provisioned on nodes k8s-worker1 and k8s-worker2, but not on k8s-worker3.Risks and caveatsCaution:It is deeply encouraged to encrypt the swap space. See Memory-backed volumes memory-backed volumes for more info.Having swap available on a system reduces predictability. While swap can enhance performance by making more RAM available, swapping data back to memory is a heavy operation, sometimes slower by many orders of magnitude, which can cause unexpected performance regressions. Furthermore, swap changes a system's behaviour under memory pressure. Enabling swap increases the risk of noisy neighbors, where Pods that frequently use their RAM may cause other Pods to swap. In addition, since swap allows for greater memory usage for workloads in Kubernetes that cannot be predictably accounted for, and due to unexpected packing configurations, the scheduler currently does not account for swap memory usage. This heightens the risk of noisy neighbors.The performance of a node with swap memory enabled depends on the underlying physical storage. When swap memory is in use, performance will be significantly worse in an I/O operations per second (IOPS) constrained environment, such as a cloud VM with I/O throttling, when compared to faster storage mediums like solid-state drives or NVMe. As swap might cause IO pressure, it is recommended to give a higher IO latency priority to system critical daemons. See the relevant section in the recommended practices section below.Memory-backed volumesOn Linux nodes, memory-backed volumes (such as secret volume mounts, or emptyDir with medium: Memory) are implemented with a tmpfs filesystem. The contents of such volumes should remain in memory at all times, hence should not be swapped to disk. To ensure the contents of such volumes remain in memory, the noswap tmpfs option is being used.The Linux kernel officially supports the noswap option from version 6.3 (more info can be found in Linux Kernel Version Requirements). However, the different distributions often choose to backport this mount option to older Linux versions as well.In order to verify whether the node supports the noswap option, the kubelet will do the following:If the kernel's version is above 6.3 then the noswap option will be assumed to be supported.Otherwise, kubelet would try to mount a dummy tmpfs with the noswap option at startup. If kubelet fails with an error indicating of an unknown option, noswap will be assumed to not be supported, hence will not be used. A kubelet log entry will be emitted to warn the user about memory-backed volumes might swap to disk. If kubelet succeeds, the dummy tmpfs will be deleted and the noswap option will be used.If the noswap option is not supported, kubelet will emit a warning log entry, then continue its execution.See the section above with an example for setting unencrypted swap. However, handling encrypted swap is not within the scope of kubelet; rather, it is a general OS configuration concern and should be addressed at that level. It is the administrator's responsibility to provision encrypted swap to mitigate this risk.EvictionsConfiguring memory eviction thresholds for swap-enabled nodes can be tricky.With swap being disabled, it is reasonable to configure kubelet's eviction thresholds to be a bit lower than the node's memory capacity. The rationale is that we want Kubernetes to start evicting Pods before the node runs out of memory and invokes the Out Of Memory (OOM) killer, since the OOM killer is not Kubernetes-aware, therefore does not consider things like QoS, pod priority, or other Kubernetes-specific factors.With swap enabled, the situation is more complex. In Linux, the vm.min_free_kbytes parameter defines the memory threshold for the kernel to start aggressively reclaiming memory, which includes swapping out pages. If the kubelet's eviction thresholds are set in a way that eviction would take place before the kernel starts reclaiming memory, it could lead to workloads never being able to swap out during node memory pressure. However, setting the eviction thresholds too high could result in the node running out of memory and invoking the OOM killer, which is not ideal either.To address this, it is recommended to set the kubelet's eviction thresholds to be slightly lower than the vm.min_free_kbytes value. This way, the node can start swapping before kubelet would start evicting Pods, allowing workloads to swap out unused data and preventing evictions from happening. On the other hand, since it is just slightly lower, kubelet is likely to start evicting Pods before the node runs out of memory, thus avoiding the OOM killer.The value of vm.min_free_kbytes can be determined by running the following command on the node:cat /proc/sys/vm/min_free_kbytes Unutilized swap spaceUnder the LimitedSwap behavior, the amount of swap available to a Pod is determined automatically, based on the proportion of the memory requested relative to the node's total memory (For more details, see the section below).This design means that usually there would be some portion of swap that will remain restricted for Kubernetes workloads. For example, since Kubernetes 1.34 does not permit swap use for Pods in the Guaranteed QoS class, the amount of swap that's proportional to the memory request for Guaranteed pods would remain unused by Kubernetes workloads.This behavior carries some risk in a situation where many pods are not eligible for swapping. On the other hand, it effectively keeps some system-reserved amount of swap memory that can be used by processes outside of Kubernetes' scope, such as system daemons and even kubelet itself.Good practice for using swap in a Kubernetes clusterDisable swap for system-critical daemonsDuring the testing phase and based on user

Cluster Administration


Runtime ClassFEATURE STATE: Kubernetes v1.20 [stable]This page describes the RuntimeClass resource and runtime selection mechanism.RuntimeClass is a feature for selecting the container runtime configuration. The container runtime configuration is used to run a Pod's containers.MotivationYou can set a different RuntimeClass between different Pods to provide a balance of performance versus security. For example, if part of your workload deserves a high level of information security assurance, you might choose to schedule those Pods so that they run in a container runtime that uses hardware virtualization. You'd then benefit from the extra isolation of the alternative runtime, at the expense of some additional overhead.You can also use RuntimeClass to run different Pods with the same container runtime but with different settings.SetupConfigure the CRI implementation on nodes (runtime dependent)Create the corresponding RuntimeClass resources1. Configure the CRI implementation on nodesThe configurations available through RuntimeClass are Container Runtime Interface (CRI) implementation dependent. See the corresponding documentation (below) for your CRI implementation for how to configure.Note:RuntimeClass assumes a homogeneous node configuration across the cluster by default (which means that all nodes are configured the same way with respect to container runtimes). To support heterogeneous node configurations, see Scheduling below.The configurations have a corresponding handler name, referenced by the RuntimeClass. The handler must be a valid DNS label name.2. Create the corresponding RuntimeClass resourcesThe configurations setup in step 1 should each have an associated handler name, which identifies the configuration. For each handler, create a corresponding RuntimeClass object.The RuntimeClass resource currently only has 2 significant fields: the RuntimeClass name (metadata.name) and the handler (handler). The object definition looks like this:# RuntimeClass is defined in the node.k8s.io API group apiVersion: node.k8s.io/v1 kind: RuntimeClass metadata: # The name the RuntimeClass will be referenced by. # RuntimeClass is a non-namespaced resource. name: myclass # The name of the corresponding CRI configuration handler: myconfiguration The name of a RuntimeClass object must be a valid DNS subdomain name.Note:It is recommended that RuntimeClass write operations (create/update/patch/delete) be restricted to the cluster administrator. This is typically the default. See Authorization Overview for more details.UsageOnce RuntimeClasses are configured for the cluster, you can specify a runtimeClassName in the Pod spec to use it. For example:apiVersion: v1 kind: Pod metadata: name: mypod spec: runtimeClassName: myclass # ... This will instruct the kubelet to use the named RuntimeClass to run this pod. If the named RuntimeClass does not exist, or the CRI cannot run the corresponding handler, the pod will enter the Failed terminal phase. Look for a corresponding event for an error message.If no runtimeClassName is specified, the default RuntimeHandler will be used, which is equivalent to the behavior when the RuntimeClass feature is disabled.CRI ConfigurationFor more details on setting up CRI runtimes, see CRI installation.containerdRuntime handlers are configured through containerd's configuration at /etc/containerd/config.toml. Valid handlers are configured under the runtimes section:[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.${HANDLER_NAME}] See containerd's config documentation for more details:CRI-ORuntime handlers are configured through CRI-O's configuration at /etc/crio/crio.conf. Valid handlers are configured under the crio.runtime table:[crio.runtime.runtimes.${HANDLER_NAME}] runtime_path = "${PATH_TO_BINARY}" See CRI-O's config documentation for more details.SchedulingFEATURE STATE: Kubernetes v1.16 [beta]By specifying the scheduling field for a RuntimeClass, you can set constraints to ensure that Pods running with this RuntimeClass are scheduled to nodes that support it. If scheduling is not set, this RuntimeClass is assumed to be supported by all nodes.To ensure pods land on nodes supporting a specific RuntimeClass, that set of nodes should have a common label which is then selected by the runtimeclass.scheduling.nodeSelector field. The RuntimeClass's nodeSelector is merged with the pod's nodeSelector in admission, effectively taking the intersection of the set of nodes selected by each. If there is a conflict, the pod will be rejected.If the supported nodes are tainted to prevent other RuntimeClass pods from running on the node, you can add tolerations to the RuntimeClass. As with the nodeSelector, the tolerations are merged with the pod's tolerations in admission, effectively taking the union of the set of nodes tolerated by each.To learn more about configuring the node selector and tolerations, see Assigning Pods to Nodes.Pod OverheadFEATURE STATE: Kubernetes v1.24 [stable]You can specify overhead resources that are associated with running a Pod. Declaring overhead allows the cluster (including the scheduler) to account for it when making decisions about Pods and resources.Pod overhead is defined in RuntimeClass through the overhead field. Through the use of this field, you can specify the overhead of running pods utilizing this RuntimeClass and ensure these overheads are accounted for in Kubernetes.What's nextRuntimeClass DesignRuntimeClass Scheduling DesignRead about the Pod Overhead conceptPodOverhead Feature Design

Runtime Class


DisruptionsThis guide is for application owners who want to build highly available applications, and thus need to understand what types of disruptions can happen to Pods.It is also for cluster administrators who want to perform automated cluster actions, like upgrading and autoscaling clusters.Voluntary and involuntary disruptionsPods do not disappear until someone (a person or a controller) destroys them, or there is an unavoidable hardware or system software error.We call these unavoidable cases involuntary disruptions to an application. Examples are:a hardware failure of the physical machine backing the nodecluster administrator deletes VM (instance) by mistakecloud provider or hypervisor failure makes VM disappeara kernel panicthe node disappears from the cluster due to cluster network partitioneviction of a pod due to the node being out-of-resources.Except for the out-of-resources condition, all these conditions should be familiar to most users; they are not specific to Kubernetes.We call other cases voluntary disruptions. These include both actions initiated by the application owner and those initiated by a Cluster Administrator. Typical application owner actions include:deleting the deployment or other controller that manages the podupdating a deployment's pod template causing a restartdirectly deleting a pod (e.g. by accident)Cluster administrator actions include:Draining a node for repair or upgrade.Draining a node from a cluster to scale the cluster down (learn about Node Autoscaling).Removing a pod from a node to permit something else to fit on that node.These actions might be taken directly by the cluster administrator, or by automation run by the cluster administrator, or by your cluster hosting provider.Ask your cluster administrator or consult your cloud provider or distribution documentation to determine if any sources of voluntary disruptions are enabled for your cluster. If none are enabled, you can skip creating Pod Disruption Budgets.Caution:Not all voluntary disruptions are constrained by Pod Disruption Budgets. For example, deleting deployments or pods bypasses Pod Disruption Budgets.Dealing with disruptionsHere are some ways to mitigate involuntary disruptions:Ensure your pod requests the resources it needs.Replicate your application if you need higher availability. (Learn about running replicated stateless and stateful applications.)For even higher availability when running replicated applications, spread applications across racks (using anti-affinity) or across zones (if using a multi-zone cluster.)The frequency of voluntary disruptions varies. On a basic Kubernetes cluster, there are no automated voluntary disruptions (only user-triggered ones). However, your cluster administrator or hosting provider may run some additional services which cause voluntary disruptions. For example, rolling out node software updates can cause voluntary disruptions. Also, some implementations of cluster (node) autoscaling may cause voluntary disruptions to defragment and compact nodes. Your cluster administrator or hosting provider should have documented what level of voluntary disruptions, if any, to expect. Certain configuration options, such as using PriorityClasses in your pod spec can also cause voluntary (and involuntary) disruptions.Pod disruption budgetsFEATURE STATE: Kubernetes v1.21 [stable]Kubernetes offers features to help you run highly available applications even when you introduce frequent voluntary disruptions.As an application owner, you can create a PodDisruptionBudget (PDB) for each application. A PDB limits the number of Pods of a replicated application that are down simultaneously from voluntary disruptions. For example, a quorum-based application would like to ensure that the number of replicas running is never brought below the number needed for a quorum. A web front end might want to ensure that the number of replicas serving load never falls below a certain percentage of the total.Cluster managers and hosting providers should use tools which respect PodDisruptionBudgets by calling the Eviction API instead of directly deleting pods or deployments.For example, the kubectl drain subcommand lets you mark a node as going out of service. When you run kubectl drain, the tool tries to evict all of the Pods on the Node you're taking out of service. The eviction request that kubectl submits on your behalf may be temporarily rejected, so the tool periodically retries all failed requests until all Pods on the target node are terminated, or until a configurable timeout is reached.A PDB specifies the number of replicas that an application can tolerate having, relative to how many it is intended to have. For example, a Deployment which has a .spec.replicas: 5 is supposed to have 5 pods at any given time. If its PDB allows for there to be 4 at a time, then the Eviction API will allow voluntary disruption of one (but not two) pods at a time.The group of pods that comprise the application is specified using a label selector, the same as the one used by the application's controller (deployment, stateful-set, etc).The "intended" number of pods is computed from the .spec.replicas of the workload resource that is managing those pods. The control plane discovers the owning workload resource by examining the .metadata.ownerReferences of the Pod.Involuntary disruptions cannot be prevented by PDBs; however they do count against the budget.Pods which are deleted or unavailable due to a rolling upgrade to an application do count against the disruption budget, but workload resources (such as Deployment and StatefulSet) are not limited by PDBs when doing rolling upgrades. Instead, the handling of failures during application updates is configured in the spec for the specific workload resource.It is recommended to set AlwaysAllow Unhealthy Pod Eviction Policy to your PodDisruptionBudgets to support eviction of misbehaving applications during a node drain. The default behavior is to wait for the application pods to become healthy before the drain can proceed.When a pod is evicted using the eviction API, it is gracefully terminated, honoring the terminationGracePeriodSeconds setting in its PodSpec.PodDisruptionBudget exampleConsider a cluster with 3 nodes, node-1 through node-3. The cluster is running several applications. One of them has 3 replicas initially called pod-a, pod-b, and pod-c. Another, unrelated pod without a PDB, called pod-x, is also shown. Initially, the pods are laid out as follows:node-1node-2node-3pod-a availablepod-b availablepod-c availablepod-x availableAll 3 pods are part of a deployment, and they collectively have a PDB which requires there be at least 2 of the 3 pods to be available at all times.For example, assume the cluster administrator wants to reboot into a new kernel version to fix a bug in the kernel. The cluster administrator first tries to drain node-1 using the kubectl drain command. That tool tries to evict pod-a and pod-x. This succeeds immediately. Both pods go into the terminating state at the same time. This puts the cluster in this state:node-1 drainingnode-2node-3pod-a terminatingpod-b availablepod-c availablepod-x terminatingThe deployment notices that one of the pods is terminating, so it creates a replacement called pod-d. Since node-1 is cordoned, it lands on another node. Something has also created pod-y as a replacement for pod-x.(Note: for a StatefulSet, pod-a, which would be called something like pod-0, would need to terminate completely before its replacement, which is also called pod-0 but has a different UID, could be created. Otherwise, the example applies to a StatefulSet as well.)Now the cluster is in this state:node-1 drainingnode-2node-3pod-a terminatingpod-b availablepod-c availablepod-x terminatingpod-d startingpod-yAt some point, the pods terminate, and the cluster looks like this:node-1 drainednode-2node-3pod-b availablepod-c availablepod-d startingpod-yAt this point, if an impatient cluster administrator tries to drain node-2 or node-3, the drain command will block, because there are only 2 available pods for the deployment, and its PDB requires at least 2. After some time passes, pod-d becomes available.The cluster state now looks like this:node-1 drainednode-2node-3pod-b availablepod-c availablepod-d availablepod-yNow, the cluster administrator tries to drain node-2. The drain command will try to evict the two pods in some order, say pod-b first and then pod-d. It will succeed at evicting pod-b. But, when it tries to evict pod-d, it will be refused because that would leave only one pod available for the deployment.The deployment creates a replacement for pod-b called pod-e. Because there are not enough resources in the cluster to schedule pod-e the drain will again block. The cluster may end up in this state:node-1 drainednode-2node-3no nodepod-b terminatingpod-c availablepod-e pendingpod-d availablepod-yAt this point, the cluster administrator needs to add a node back to the cluster to proceed with the upgrade.You can see how Kubernetes varies the rate at which disruptions can happen, according to:how many replicas an application needshow long it takes to gracefully shutdown an instancehow long it takes a new instance to start upthe type of controllerthe cluster's resource capacityPod disruption conditionsFEATURE STATE: Kubernetes v1.31 [stable] (enabled by default: true)A dedicated Pod DisruptionTarget condition is added to indicate that the Pod is about to be deleted due to a disruption. The reason field of the condition additionally indicates one of the following reasons for the Pod termination:PreemptionBySchedulerPod is due to be preempted by a scheduler in order to accommodate a new Pod with a higher priority. For more information, see Pod priority preemption.DeletionByTaintManagerPod is due to be deleted by Taint Manager (which is part of the node lifecycle controller within kube-controller-manager) due to a NoExecute taint that the Pod does not tolerate; see taint-based evictions.EvictionByEvictionAPIPod has been marked for eviction using the Kubernetes API .DeletionByPodGCPod, that is bound to a no longer existing Node, is due to be deleted by Pod garbage collection.TerminationByKubeletPod has been terminated by the kubelet, because of either node pressure eviction, the graceful node shutdown, or preemption for system critical pods.In all other disruption scenarios, like eviction due to exceeding Pod container limits, Pods don't receive the DisruptionTarget condition because the disruptions were probably caused by the Pod and would reoccur on retry.Note:A Pod disruption might be interrupted. The control plane might re-attempt to continue the disruption of the same Pod, but it is not guaranteed. As a result, the DisruptionTarget condition might be added to a Pod, but that Pod might then not actually be deleted. In such a situation, after some time, the Pod disruption condition will be cleared.Along with cleaning up the pods, the Pod garbage collector (PodGC) will also mark them as failed if they are in a non-terminal phase (see also Pod garbage collection).When using a Job (or CronJob), you may want to use these Pod disruption conditions as part of your Job's Pod failure policy.Separating Cluster Owner and Application Owner RolesOften, it is useful to think of the Cluster Manager and Application Owner as separate roles with limited knowledge of each other. This separation of responsibilities may make sense in these scenarios:when there are many application teams sharing a Kubernetes cluster, and there is natural specialization of roleswhen third-party tools or services are used to automate cluster managementPod Disruption Budgets support this separation of roles by providing an interface between the roles.If you do not have such a separation of responsibilities in your organization, you may not need to use Pod Disruption Budgets.How to perform Disruptive Actions on your ClusterIf you are a Cluster Administrator, and you need to perform a disruptive action on all the nodes in your cluster, such as a node or system software upgrade, here are some options:Accept downtime during the upgrade.Failover to another complete replica cluster.No downtime, but may be costly both for the duplicated nodes and for human effort to orchestrate the switchover.Write disruption tolerant applications and use PDBs.No downtime.Minimal resource duplication.Allows more automation of cluster administration.Writing disruption-tolerant applications is tricky, but the work to tolerate voluntary disruptions largely overlaps with work to support autoscaling and tolerating involuntary disruptions.What's nextFollow steps to protect your application by configuring a Pod Disruption Budget.Learn more about draining nodesLearn about updating a deployment including steps to maintain its availability during the rollout.

Disruptions


Using Minikube to Create a ClusterObjectivesLearn what a Kubernetes cluster is.Learn what Minikube is.Start a Kubernetes cluster on your computer.Kubernetes ClustersKubernetes is a production-grade, open-source platform that orchestrates the placement (scheduling) and execution of application containers within and across computer clusters.Kubernetes coordinates a highly available cluster of computers that are connected to work as a single unit. The abstractions in Kubernetes allow you to deploy containerized applications to a cluster without tying them specifically to individual machines. To make use of this new model of deployment, applications need to be packaged in a way that decouples them from individual hosts: they need to be containerized. Containerized applications are more flexible and available than in past deployment models, where applications were installed directly onto specific machines as packages deeply integrated into the host. Kubernetes automates the distribution and scheduling of application containers across a cluster in a more efficient way. Kubernetes is an open-source platform and is production-ready.A Kubernetes cluster consists of two types of resources:The Control Plane coordinates the clusterNodes are the workers that run applicationsCluster DiagramThe Control Plane is responsible for managing the cluster. The Control Plane coordinates all activities in your cluster, such as scheduling applications, maintaining applications' desired state, scaling applications, and rolling out new updates.Control Planes manage the cluster and the nodes that are used to host the running applications.A node is a VM or a physical computer that serves as a worker machine in a Kubernetes cluster. Each node has a Kubelet, which is an agent for managing the node and communicating with the Kubernetes control plane. The node should also have tools for handling container operations, such as containerd or CRI-O. A Kubernetes cluster that handles production traffic should have a minimum of three nodes because if one node goes down, both an etcd member and a control plane instance are lost, and redundancy is compromised. You can mitigate this risk by adding more control plane nodes.When you deploy applications on Kubernetes, you tell the control plane to start the application containers. The control plane schedules the containers to run on the cluster's nodes. Node-level components, such as the kubelet, communicate with the control plane using the Kubernetes API, which the control plane exposes. End users can also use the Kubernetes API directly to interact with the cluster.A Kubernetes cluster can be deployed on either physical or virtual machines. To get started with Kubernetes development, you can use Minikube. Minikube is a lightweight Kubernetes implementation that creates a VM on your local machine and deploys a simple cluster containing only one node. Minikube is available for Linux, macOS, and Windows systems. The Minikube CLI provides basic bootstrapping operations for working with your cluster, including start, stop, status, and delete.What's nextTutorial Hello Minikube.Learn more about Cluster Architecture.

Using Minikube to Create a Cluster