# Authorization

AuthorizationDetails of Kubernetes authorization mechanisms and supported authorization modes.Kubernetes authorization takes place following authentication. Usually, a client making a request must be authenticated (logged in) before its request can be allowed; however, Kubernetes also allows anonymous requests in some circumstances.For an overview of how authorization fits into the wider context of API access control, read Controlling Access to the Kubernetes API.Authorization verdictsKubernetes authorization of API requests takes place within the API server. The API server evaluates all of the request attributes against all policies, potentially also consulting external services, and then allows or denies the request.All parts of an API request must be allowed by some authorization mechanism in order to proceed. In other words: access is denied by default.Note:Access controls and policies that depend on specific fields of specific kinds of objects are handled by admission controllers.Kubernetes admission control happens after authorization has completed (and, therefore, only when the authorization decision was to allow the request).When multiple authorization modules are configured, each is checked in sequence. If any authorizer approves or denies a request, that decision is immediately returned and no other authorizer is consulted. If all modules have no opinion on the request, then the request is denied. An overall deny verdict means that the API server rejects the request and responds with an HTTP 403 (Forbidden) status.Request attributes used in authorizationKubernetes reviews only the following API request attributes:user - The user string provided during authentication.group - The list of group names to which the authenticated user belongs.extra - A map of arbitrary string keys to string values, provided by the authentication layer.API - Indicates whether the request is for an API resource.Request path - Path to miscellaneous non-resource endpoints like /api or /healthz.API request verb - API verbs like get, list, create, update, patch, watch, delete, and deletecollection are used for resource requests. To determine the request verb for a resource API endpoint, see request verbs and authorization.HTTP request verb - Lowercased HTTP methods like get, post, put, and delete are used for non-resource requests.Resource - The ID or name of the resource that is being accessed (for resource requests only) -- For resource requests using get, update, patch, and delete verbs, you must provide the resource name.Subresource - The subresource that is being accessed (for resource requests only).Namespace - The namespace of the object that is being accessed (for namespaced resource requests only).API group - The API Group being accessed (for resource requests only). An empty string designates the core API group.Request verbs and authorizationNon-resource requestsRequests to endpoints other than /api/v1/... or /apis/<group>/<version>/... are considered non-resource requests, and use the lower-cased HTTP method of the request as the verb. For example, making a GET request using HTTP to endpoints such as /api or /healthz would use get as the verb.Resource requestsTo determine the request verb for a resource API endpoint, Kubernetes maps the HTTP verb used and considers whether or not the request acts on an individual resource or on a collection of resources:HTTP verbrequest verbPOSTcreateGET, HEADget (for individual resources), list (for collections, including full object content), watch (for watching an individual resource or collection of resources)PUTupdatePATCHpatchDELETEdelete (for individual resources), deletecollection (for collections)Caution:+The get, list and watch verbs can all return the full details of a resource. In terms of access to the returned data they are equivalent. For example, list on secrets will reveal the data attributes of any returned resources.Kubernetes sometimes checks authorization for additional permissions using specialized verbs. For example:Special cases of authenticationimpersonate verb on users, groups, and serviceaccounts in the core API group, and the userextras in the authentication.k8s.io API group.Authorization of CertificateSigningRequestsapprove verb for CertificateSigningRequests, and update for revisions to existing approvalsRBACbind and escalate verbs on roles and clusterroles resources in the rbac.authorization.k8s.io API group.Authorization contextKubernetes expects attributes that are common to REST API requests. This means that Kubernetes authorization works with existing organization-wide or cloud-provider-wide access control systems which may handle other APIs besides the Kubernetes API.Authorization modesThe Kubernetes API server may authorize a request using one of several authorization modes:AlwaysAllowThis mode allows all requests, which brings security risks. Use this authorization mode only if you do not require authorization for your API requests (for example, for testing).AlwaysDenyThis mode blocks all requests. Use this authorization mode only for testing.ABAC (attribute-based access control)Kubernetes ABAC mode defines an access control paradigm whereby access rights are granted to users through the use of policies which combine attributes together. The policies can use any type of attributes (user attributes, resource attributes, object, environment attributes, etc).RBAC (role-based access control)Kubernetes RBAC is a method of regulating access to computer or network resources based on the roles of individual users within an enterprise. In this context, access is the ability of an individual user to perform a specific task, such as view, create, or modify a file.In this mode, Kubernetes uses the rbac.authorization.k8s.io API group to drive authorization decisions, allowing you to dynamically configure permission policies through the Kubernetes API.NodeA special-purpose authorization mode that grants permissions to kubelets based on the pods they are scheduled to run. To learn more about the Node authorization mode, see Node Authorization.WebhookKubernetes webhook mode for authorization makes a synchronous HTTP callout, blocking the request until the remote HTTP service responds to the query.You can write your own software to handle the callout, or use solutions from the ecosystem.Warning:Enabling the AlwaysAllow mode bypasses authorization; do not use this on a cluster where you do not trust all potential API clients, including the workloads that you run.Authorization mechanisms typically return either a deny or no opinion result; see authorization verdicts for more on this. Activating the AlwaysAllow means that if all other authorizers return “no opinion”, the request is allowed. For example, --authorization-mode=AlwaysAllow,RBAC has the same effect as --authorization-mode=AlwaysAllow because Kubernetes RBAC does not provide negative (deny) access rules.You should not use the AlwaysAllow mode on a Kubernetes cluster where the API server is reachable from the public internet.The system:masters groupThe system:masters group is a built-in Kubernetes group that grants unrestricted access to the API server. Any user assigned to this group has full cluster administrator privileges, bypassing any authorization restrictions imposed by the RBAC or Webhook mechanisms. Avoid adding users to this group. If you do need to grant a user cluster-admin rights, you can create a ClusterRoleBinding to the built-in cluster-admin ClusterRole.Authorization mode configurationYou can configure the Kubernetes API server's authorizer chain using either a configuration file only or command line arguments.You have to pick one of the two configuration approaches; setting both --authorization-config path and configuring an authorization webhook using the --authorization-mode and --authorization-webhook-* command line arguments is not allowed. If you try this, the API server reports an error message during startup, then exits immediately.Configuring the API Server using an authorization config fileFEATURE STATE: Kubernetes v1.32 [stable] (enabled by default: true)Kubernetes lets you configure authorization chains that can include multiple webhooks. The authorization items in that chain can have well-defined parameters that validate requests in a particular order, offering you fine-grained control, such as explicit Deny on failures.The configuration file approach even allows you to specify CEL rules to pre-filter requests before they are dispatched to webhooks, helping you to prevent unnecessary invocations. The API server also automatically reloads the authorizer chain when the configuration file is modified.You specify the path to the authorization configuration using the --authorization-config command line argument.If you want to use command line arguments instead of a configuration file, that's also a valid and supported approach. Some authorization capabilities (for example: multiple webhooks, webhook failure policy, and pre-filter rules) are only available if you use an authorization configuration file.Example configuration--- # # DO NOT USE THE CONFIG AS IS. THIS IS AN EXAMPLE. # apiVersion: apiserver.config.k8s.io/v1 kind: AuthorizationConfiguration authorizers: - type: Webhook # Name used to describe the authorizer # This is explicitly used in monitoring machinery for metrics # Note: # - Validation for this field is similar to how K8s labels are validated today. # Required, with no default name: webhook webhook: # The duration to cache 'authorized' responses from the webhook # authorizer. # Same as setting `--authorization-webhook-cache-authorized-ttl` flag # Default: 5m0s authorizedTTL: 30s # If set to false, 'authorized' responses from the webhook are not cached # and the specified authorizedTTL is ignored/has no effect. # Same as setting `--authorization-webhook-cache-authorized-ttl` flag to `0`. # Note: Setting authorizedTTL to `0` results in its default value being used. # Default: true cacheAuthorizedRequests: true # The duration to cache 'unauthorized' responses from the webhook # authorizer. # Same as setting `--authorization-webhook-cache-unauthorized-ttl` flag # Default: 30s unauthorizedTTL: 30s # If set to false, 'unauthorized' responses from the webhook are not cached # and the specified unauthorizedTTL is ignored/has no effect. # Same as setting `--authorization-webhook-cache-unauthorized-ttl` flag to `0`. # Note: Setting unauthorizedTTL to `0` results in its default value being used. # Default: true cacheUnauthorizedRequests: true # Timeout for the webhook request # Maximum allowed is 30s. # Required, with no default. timeout: 3s # The API version of the authorization.k8s.io SubjectAccessReview to # send to and expect from the webhook. # Same as setting `--authorization-webhook-version` flag # Required, with no default # Valid values: v1beta1, v1 subjectAccessReviewVersion: v1 # MatchConditionSubjectAccessReviewVersion specifies the SubjectAccessReview # version the CEL expressions are evaluated against # Valid values: v1 # Required, no default value matchConditionSubjectAccessReviewVersion: v1 # Controls the authorization decision when a webhook request fails to # complete or returns a malformed response or errors evaluating # matchConditions. # Valid values: # - NoOpinion: continue to subsequent authorizers to see if one of # them allows the request # - Deny: reject the request without consulting subsequent authorizers # Required, with no default. failurePolicy: Deny connectionInfo: # Controls how the webhook should communicate with the server. # Valid values: # - KubeConfigFile: use the file specified in kubeConfigFile to locate the # server. # - InClusterConfig: use the in-cluster configuration to call the # SubjectAccessReview API hosted by kube-apiserver. This mode is not # allowed for kube-apiserver. type: KubeConfigFile # Path to KubeConfigFile for connection info # Required, if connectionInfo.Type is KubeConfigFile kubeConfigFile: /kube-system-authz-webhook.yaml # matchConditions is a list of conditions that must be met for a request to be sent to this # webhook. An empty list of matchConditions matches all requests. # There are a maximum of 64 match conditions allowed. # # The exact matching logic is (in order): # 1. If at least one matchCondition evaluates to FALSE, then the webhook is skipped. # 2. If ALL matchConditions evaluate to TRUE, then the webhook is called. # 3. If at least one matchCondition evaluates to an error (but none are FALSE): # - If failurePolicy=Deny, then the webhook rejects the request # - If failurePolicy=NoOpinion, then the error is ignored and the webhook is skipped matchConditions: # expression represents the expression which will be evaluated by CEL. Must evaluate to bool. # CEL expressions have access to the contents of the SubjectAccessReview in v1 version. # If version specified by subjectAccessReviewVersion in the request variable is v1beta1, # the contents would be converted to the v1 version before evaluating the CEL expression. # # Documentation on CEL: https://kubernetes.io/docs/reference/using-api/cel/ # # only send resource requests to the webhook - expression: has(request.resourceAttributes) # only intercept requests to kube-system - expression: request.resourceAttributes.namespace == 'kube-system' # don't intercept requests from kube-system service accounts - expression: "!('system:serviceaccounts:kube-system' in request.groups)" - type: Node name: node - type: RBAC name: rbac - type: Webhook name: in-cluster-authorizer webhook: authorizedTTL: 5m unauthorizedTTL: 30s timeout: 3s subjectAccessReviewVersion: v1 failurePolicy: NoOpinion connectionInfo: type: InClusterConfigWhen configuring the authorizer chain using a configuration file, make sure all the control plane nodes have the same file contents. Take a note of the API server configuration when upgrading / downgrading your clusters. For example, if upgrading from Kubernetes 1.33 to Kubernetes 1.34, you would need to make sure the config file is in a format that Kubernetes 1.34 can understand, before you upgrade the cluster. If you downgrade to 1.33, you would need to set the configuration appropriately.Authorization configuration and reloadsKubernetes reloads the authorization configuration file when the API server observes a change to the file, and also on a 60 second schedule if no change events were observed.Note:You must ensure that all non-webhook authorizer types remain unchanged in the file on reload.A reload must not add or remove Node or RBAC authorizers (they can be reordered, but cannot be added or removed).Command line authorization mode configurationYou can use the following modes:--authorization-mode=ABAC (Attribute-based access control mode)--authorization-mode=RBAC (Role-based access control mode)--authorization-mode=Node (Node authorizer)--authorization-mode=Webhook (Webhook authorization mode)--authorization-mode=AlwaysAllow (always allows requests; carries security risks)--authorization-mode=AlwaysDeny (always denies requests)You can choose more than one authorization mode; for example: --authorization-mode=Node,RBAC,WebhookKubernetes checks authorization modules based on the order that you specify them on the API server's command line, so an earlier module has higher priority to allow or deny a request.You cannot combine the --authorization-mode command line argument with the --authorization-config command line argument used for configuring authorization using a local file.For more information on command line arguments to the API server, read the kube-apiserver reference.Privilege escalation via workload creation or editsUsers who can create/edit pods in a namespace, either directly or through an object that enables indirect workload management, may be able to escalate their privileges in that namespace. The potential routes to privilege escalation include Kubernetes API extensions and their associated controllers.Caution:As a cluster administrator, use caution when granting access to create or edit workloads. Some details of how these can be misused are documented in escalation paths.Escalation pathsThere are different ways that an attacker or untrustworthy user could gain additional privilege within a namespace, if you allow them to run arbitrary Pods in that namespace:Mounting arbitrary Secrets in that namespaceCan be used to access confidential information meant for other workloadsCan be used to obtain a more privileged ServiceAccount's service account tokenUsing arbitrary ServiceAccounts in that namespaceCan perform Kubernetes API actions as another workload (impersonation)Can perform any privileged actions that ServiceAccount hasMounting or using ConfigMaps meant for other workloads in that namespaceCan be used to obtain information meant for other workloads, such as database host names.Mounting volumes meant for other workloads in that namespaceCan be used to obtain information meant for other workloads, and change it.Caution:As a system administrator, you should be cautious when deploying CustomResourceDefinitions that let users make changes to the above areas. These may open privilege escalations paths. Consider the consequences of this kind of change when deciding on your authorization controls.Checking API accesskubectl provides the auth can-i subcommand for quickly querying the API authorization layer. The command uses the SelfSubjectAccessReview API to determine if the current user can perform a given action, and works regardless of the authorization mode used.kubectl auth can-i create deployments --namespace dev The output is similar to this:yes kubectl auth can-i create deployments --namespace prod The output is similar to this:no Administrators can combine this with user impersonation to determine what action other users can perform.kubectl auth can-i list secrets --namespace dev --as dave The output is similar to this:no Similarly, to check whether a ServiceAccount named dev-sa in Namespace dev can list Pods in the Namespace target:kubectl auth can-i list pods \ --namespace target \ --as system:serviceaccount:dev:dev-sa The output is similar to this:yes SelfSubjectAccessReview is part of the authorization.k8s.io API group, which exposes the API server authorization to external services. Other resources in this group include:SubjectAccessReviewAccess review for any user, not only the current one. Useful for delegating authorization decisions to the API server. For example, the kubelet and extension API servers use this to determine user access to their own APIs.LocalSubjectAccessReviewLike SubjectAccessReview but restricted to a specific namespace.SelfSubjectRulesReviewA review which returns the set of actions a user can perform within a namespace. Useful for users to quickly summarize their own access, or for UIs to hide/show actions.These APIs can be queried by creating normal Kubernetes resources, where the response status field of the returned object is the result of the query. For example:kubectl create -f - -o yaml << EOF apiVersion: authorization.k8s.io/v1 kind: SelfSubjectAccessReview spec: resourceAttributes: group: apps resource: deployments verb: create namespace: dev EOF The generated SelfSubjectAccessReview is similar to:apiVersion: authorization.k8s.io/v1 kind: SelfSubjectAccessReview metadata: creationTimestamp: null spec: resourceAttributes: group: apps resource: deployments namespace: dev verb: create status: allowed: true denied: falseWhat's nextTo learn more about Authentication, see Authentication.For an overview, read Controlling Access to the Kubernetes API.To learn more about Admission Control, see Using Admission Controllers.Read more about Common Expression Language in Kubernetes.


# Learn Kubernetes Basics

This is the multi-page printable view of this section. Click here to print.Return to the regular view of this page.Learn Kubernetes Basics1: Create a Cluster1.1: Using Minikube to Create a Cluster2: Deploy an App2.1: Using kubectl to Create a Deployment3: Explore Your App3.1: Viewing Pods and Nodes4: Expose Your App Publicly4.1: Using a Service to Expose Your App5: Scale Your App5.1: Running Multiple Instances of Your App6: Update Your App6.1: Performing a Rolling UpdateObjectivesThis tutorial provides a walkthrough of the basics of the Kubernetes cluster orchestration system. Each module contains some background information on major Kubernetes features and concepts, and a tutorial for you to follow along.Using the tutorials, you can learn to:Deploy a containerized application on a cluster.Scale the deployment.Update the containerized application with a new software version.Debug the containerized application.What can Kubernetes do for you?With modern web services, users expect applications to be available 24/7, and developers expect to deploy new versions of those applications several times a day. Containerization helps package software to serve these goals, enabling applications to be released and updated without downtime. Kubernetes helps you make sure those containerized applications run where and when you want, and helps them find the resources and tools they need to work. Kubernetes is a production-ready, open source platform designed with Google's accumulated experience in container orchestration, combined with best-of-breed ideas from the community.Kubernetes Basics Modules1. Create a Kubernetes cluster2. Deploy an app3. Explore your app4. Expose your app publicly5. Scale up your app6. Update your appWhat's


# Tutorials

This is the multi-page printable view of this section. Click here to print.Return to the regular view of this page.Tutorials1: Hello Minikube2: Learn Kubernetes Basics2.1: Create a Cluster2.1.1: Using Minikube to Create a Cluster2.2: Deploy an App2.2.1: Using kubectl to Create a Deployment2.3: Explore Your App2.3.1: Viewing Pods and Nodes2.4: Expose Your App Publicly2.4.1: Using a Service to Expose Your App2.5: Scale Your App2.5.1: Running Multiple Instances of Your App2.6: Update Your App2.6.1: Performing a Rolling Update3: Configuration3.1: Updating Configuration via a ConfigMap3.2: Configuring Redis using a ConfigMap3.3: Adopting Sidecar Containers4: Security4.1: Apply Pod Security Standards at the Cluster Level4.2: Apply Pod Security Standards at the Namespace Level4.3: Restrict a Container's Access to Resources with AppArmor4.4: Restrict a Container's Syscalls with seccomp5: Stateless Applications5.1: Exposing an External IP Address to Access an Application in a Cluster5.2: Example: Deploying PHP Guestbook application with Redis6: Stateful Applications6.1: StatefulSet Basics6.2: Example: Deploying WordPress and MySQL with Persistent Volumes6.3: Example: Deploying Cassandra with a StatefulSet6.4: Running ZooKeeper, A Distributed System Coordinator7: Cluster Management7.1: Running Kubelet in Standalone Mode7.2: Configuring swap memory on Kubernetes nodes7.3: Install Drivers and Allocate Devices with DRA7.4: Namespaces Walkthrough8: Services8.1: Connecting Applications with Services8.2: Using Source IP8.3: Explore Termination Behavior for Pods And Their EndpointsThis section of the Kubernetes documentation contains tutorials. A tutorial shows how to accomplish a goal that is larger than a single task. Typically a tutorial has several sections, each of which has a sequence of steps. Before walking through each tutorial, you may want to bookmark the Standardized Glossary page for later references.BasicsKubernetes Basics is an in-depth interactive tutorial that helps you understand the Kubernetes system and try out some basic Kubernetes features.Introduction to Kubernetes (edX)Hello MinikubeConfigurationConfiguring Redis Using a ConfigMapAuthoring PodsAdopting Sidecar ContainersStateless ApplicationsExposing an External IP Address to Access an Application in a ClusterExample: Deploying PHP Guestbook application with RedisStateful ApplicationsStatefulSet BasicsExample: WordPress and MySQL with Persistent VolumesExample: Deploying Cassandra with Stateful SetsRunning ZooKeeper, A CP Distributed SystemServicesConnecting Applications with ServicesUsing Source IPSecurityApply Pod Security Standards at Cluster levelApply Pod Security Standards at Namespace levelRestrict a Container's Access to Resources with AppArmorSeccompCluster ManagementConfiguring Swap Memory on Kubernetes NodesRunning Kubelet in Standalone ModeInstall Drivers and Allocate Devices with DRAWhat's


# Stateful Applications

This is the multi-page printable view of this section. Click here to print.Return to the regular view of this page.Stateful Applications1: StatefulSet Basics2: Example: Deploying WordPress and MySQL with Persistent Volumes3: Example: Deploying Cassandra with a StatefulSet4: Running ZooKeeper, A Distributed System Coordinator1 - StatefulSet BasicsThis tutorial provides an introduction to managing applications with StatefulSets. It demonstrates how to create, delete, scale, and update the Pods of StatefulSets.Before you beginBefore you begin this tutorial, you should familiarize yourself with the following Kubernetes concepts:PodsCluster DNSHeadless ServicesPersistentVolumesPersistentVolume ProvisioningThe kubectl command line toolYou need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. It is recommended to run this tutorial on a cluster with at least two nodes that are not acting as control plane hosts. If you do not already have a cluster, you can create one by using minikube or you can use one of these Kubernetes playgrounds:iximiuz LabsKillercodaKodeKloudPlay with KubernetesYou should configure kubectl to use a context that uses the default namespace. If you are using an existing cluster, make sure that it's OK to use that cluster's default namespace to practice. Ideally, practice in a cluster that doesn't run any real workloads.It's also useful to read the concept page about StatefulSets.Note:This tutorial assumes that your cluster is configured to dynamically provision PersistentVolumes. You'll also need to have a default StorageClass. If your cluster is not configured to provision storage dynamically, you will have to manually provision two 1 GiB volumes prior to starting this tutorial and set up your cluster so that those PersistentVolumes map to the PersistentVolumeClaim templates that the StatefulSet defines.ObjectivesStatefulSets are intended to be used with stateful applications and distributed systems. However, the administration of stateful applications and distributed systems on Kubernetes is a broad, complex topic. In order to demonstrate the basic features of a StatefulSet, and not to conflate the former topic with the latter, you will deploy a simple web application using a StatefulSet.After this tutorial, you will be familiar with the following.How to create a StatefulSetHow a StatefulSet manages its PodsHow to delete a StatefulSetHow to scale a StatefulSetHow to update a StatefulSet's PodsCreating a StatefulSetBegin by creating a StatefulSet (and the Service that it relies upon) using the example below. It is similar to the example presented in the StatefulSets concept. It creates a headless Service, nginx, to publish the IP addresses of Pods in the StatefulSet, web.application/web/web.yaml apiVersion: v1 kind: Service metadata: name: nginx labels: app: nginx spec: ports: - port: 80 name: web clusterIP: None selector: app: nginx --- apiVersion: apps/v1 kind: StatefulSet metadata: name: web spec: serviceName: "nginx" replicas: 2 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - name: nginx image: registry.k8s.io/nginx-slim:0.21 ports: - containerPort: 80 name: web volumeMounts: - name: www mountPath: /usr/share/nginx/html volumeClaimTemplates: - metadata: name: www spec: accessModes: [ "ReadWriteOnce" ] resources: requests: storage: 1Gi You will need to use at least two terminal windows. In the first terminal, use kubectl get to watch the creation of the StatefulSet's Pods.# use this terminal to run commands that specify --watch # end this watch when you are asked to start a new watch kubectl get pods --watch -l app=nginx In the second terminal, use kubectl apply to create the headless Service and StatefulSet:kubectl apply -f https://k8s.io/examples/application/web/web.yaml service/nginx created statefulset.apps/web created The command above creates two Pods, each running an NGINX webserver. Get the nginx Service...kubectl get service nginx NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE nginx ClusterIP None <none> 80/TCP 12s ...then get the web StatefulSet, to verify that both were created successfully:kubectl get statefulset web NAME READY AGE web 2/2 37s Ordered Pod creationA StatefulSet defaults to creating its Pods in a strict order.For a StatefulSet with n replicas, when Pods are being deployed, they are created sequentially, ordered from {0..n-1}. Examine the output of the kubectl get command in the first terminal. Eventually, the output will look like the example below.# Do not start a new watch; # this should already be running kubectl get pods --watch -l app=nginx NAME READY STATUS RESTARTS AGE web-0 0/1 Pending 0 0s web-0 0/1 Pending 0 0s web-0 0/1 ContainerCreating 0 0s web-0 1/1 Running 0 19s web-1 0/1 Pending 0 0s web-1 0/1 Pending 0 0s web-1 0/1 ContainerCreating 0 0s web-1 1/1 Running 0 18s Notice that the web-1 Pod is not launched until the web-0 Pod is Running (see Pod Phase) and Ready (see type in Pod Conditions).Later in this tutorial you will practice parallel startup.Note:To configure the integer ordinal assigned to each Pod in a StatefulSet, see Start ordinal.Pods in a StatefulSetPods in a StatefulSet have a unique ordinal index and a stable network identity.Examining the Pod's ordinal indexGet the StatefulSet's Pods:kubectl get pods -l app=nginx NAME READY STATUS RESTARTS AGE web-0 1/1 Running 0 1m web-1 1/1 Running 0 1m As mentioned in the StatefulSets concept, the Pods in a StatefulSet have a sticky, unique identity. This identity is based on a unique ordinal index that is assigned to each Pod by the StatefulSet controller.The Pods' names take the form <statefulset name>-<ordinal index>. Since the web StatefulSet has two replicas, it creates two Pods, web-0 and web-1.Using stable network identitiesEach Pod has a stable hostname based on its ordinal index. Use kubectl exec to execute the hostname command in each Pod:for i in 0 1; do kubectl exec "web-$i" -- sh -c 'hostname'; done web-0 web-1 Use kubectl run to execute a container that provides the nslookup command from the dnsutils package. Using nslookup on the Pods' hostnames, you can examine their in-cluster DNS addresses:kubectl run -i --tty --image busybox:1.28 dns-test --restart=Never --rm which starts a new shell. In that new shell, run:# Run this in the dns-test container shell nslookup web-0.nginx The output is similar to:Server: 10.0.0.10 Address 1: 10.0.0.10 kube-dns.kube-system.svc.cluster.local Name: web-0.nginx Address 1: 10.244.1.6 nslookup web-1.nginx Server: 10.0.0.10 Address 1: 10.0.0.10 kube-dns.kube-system.svc.cluster.local Name: web-1.nginx Address 1: 10.244.2.6 (and now exit the container shell: exit)The CNAME of the headless service points to SRV records (one for each Pod that is Running and Ready). The SRV records point to A record entries that contain the Pods' IP addresses.In one terminal, watch the StatefulSet's Pods:# Start a new watch # End this watch when you've seen that the delete is finished kubectl get pod --watch -l app=nginx In a second terminal, use kubectl delete to delete all the Pods in the StatefulSet:kubectl delete pod -l app=nginx pod "web-0" deleted pod "web-1" deleted Wait for the StatefulSet to restart them, and for both Pods to transition to Running and Ready:# This should already be running kubectl get pod --watch -l app=nginx NAME READY STATUS RESTARTS AGE web-0 0/1 ContainerCreating 0 0s NAME READY STATUS RESTARTS AGE web-0 1/1 Running 0 2s web-1 0/1 Pending 0 0s web-1 0/1 Pending 0 0s web-1 0/1 ContainerCreating 0 0s web-1 1/1 Running 0 34s Use kubectl exec and kubectl run to view the Pods' hostnames and in-cluster DNS entries. First, view the Pods' hostnames:for i in 0 1; do kubectl exec web-$i -- sh -c 'hostname'; done web-0 web-1 then, run:kubectl run -i --tty --image busybox:1.28 dns-test --restart=Never --rm which starts a new shell.In that new shell, run:# Run this in the dns-test container shell nslookup web-0.nginx The output is similar to:Server: 10.0.0.10 Address 1: 10.0.0.10 kube-dns.kube-system.svc.cluster.local Name: web-0.nginx Address 1: 10.244.1.7 nslookup web-1.nginx Server: 10.0.0.10 Address 1: 10.0.0.10 kube-dns.kube-system.svc.cluster.local Name: web-1.nginx Address 1: 10.244.2.8 (and now exit the container shell: exit)The Pods' ordinals, hostnames, SRV records, and A record names have not changed, but the IP addresses associated with the Pods may have changed. In the cluster used for this tutorial, they have. This is why it is important not to configure other applications to connect to Pods in a StatefulSet by the IP address of a particular Pod (it is OK to connect to Pods by resolving their hostname).Discovery for specific Pods in a StatefulSetIf you need to find and connect to the active members of a StatefulSet, you should query the CNAME of the headless Service (nginx.default.svc.cluster.local). The SRV records associated with the CNAME will contain only the Pods in the StatefulSet that are Running and Ready.If your application already implements connection logic that tests for liveness and readiness, you can use the SRV records of the Pods ( web-0.nginx.default.svc.cluster.local, web-1.nginx.default.svc.cluster.local), as they are stable, and your application will be able to discover the Pods' addresses when they transition to Running and Ready.If your application wants to find any healthy Pod in a StatefulSet, and therefore does not need to track each specific Pod, you could also connect to the IP address of a type: ClusterIP Service, backed by the Pods in that StatefulSet. You can use the same Service that tracks the StatefulSet (specified in the serviceName of the StatefulSet) or a separate Service that selects the right set of Pods.Writing to stable storageGet the PersistentVolumeClaims for web-0 and web-1:kubectl get pvc -l app=nginx The output is similar to:NAME STATUS VOLUME CAPACITY ACCESSMODES AGE www-web-0 Bound pvc-15c268c7-b507-11e6-932f-42010a800002 1Gi RWO 48s www-web-1 Bound pvc-15c79307-b507-11e6-932f-42010a800002 1Gi RWO 48s The StatefulSet controller created two PersistentVolumeClaims that are bound to two PersistentVolumes.As the cluster used in this tutorial is configured to dynamically provision PersistentVolumes, the PersistentVolumes were created and bound automatically.The NGINX webserver, by default, serves an index file from /usr/share/nginx/html/index.html. The volumeMounts field in the StatefulSet's spec ensures that the /usr/share/nginx/html directory is backed by a PersistentVolume.Write the Pods' hostnames to their index.html files and verify that the NGINX webservers serve the hostnames:for i in 0 1; do kubectl exec "web-$i" -- sh -c 'echo "$(hostname)" > /usr/share/nginx/html/index.html'; done for i in 0 1; do kubectl exec -i -t "web-$i" -- curl http://localhost/; done web-0 web-1 Note:If you instead see 403 Forbidden responses for the above curl command, you will need to fix the permissions of the directory mounted by the volumeMounts (due to a bug when using hostPath volumes), by running:for i in 0 1; do kubectl exec web-$i -- chmod 755 /usr/share/nginx/html; donebefore retrying the curl command above.In one terminal, watch the StatefulSet's Pods:# End this watch when you've reached the end of the section. # At the start of "Scaling a StatefulSet" you'll start a new watch. kubectl get pod --watch -l app=nginx In a second terminal, delete all of the StatefulSet's Pods:kubectl delete pod -l app=nginx pod "web-0" deleted pod "web-1" deleted Examine the output of the kubectl get command in the first terminal, and wait for all of the Pods to transition to Running and Ready.# This should already be running kubectl get pod --watch -l app=nginx NAME READY STATUS RESTARTS AGE web-0 0/1 ContainerCreating 0 0s NAME READY STATUS RESTARTS AGE web-0 1/1 Running 0 2s web-1 0/1 Pending 0 0s web-1 0/1 Pending 0 0s web-1 0/1 ContainerCreating 0 0s web-1 1/1 Running 0 34s Verify the web servers continue to serve their hostnames:for i in 0 1; do kubectl exec -i -t "web-$i" -- curl http://localhost/; done web-0 web-1 Even though web-0 and web-1 were rescheduled, they continue to serve their hostnames because the PersistentVolumes associated with their PersistentVolumeClaims are remounted to their volumeMounts. No matter what node web-0and web-1 are scheduled on, their PersistentVolumes will be mounted to the appropriate mount points.Scaling a StatefulSetScaling a StatefulSet refers to increasing or decreasing the number of replicas (horizontal scaling). This is accomplished by updating the replicas field. You can use either kubectl scale or kubectl patch to scale a StatefulSet.Scaling upScaling up means adding more replicas. Provided that your app is able to distribute work across the StatefulSet, the new larger set of Pods can perform more of that work.In one terminal window, watch the Pods in the StatefulSet:# If you already have a watch running, you can continue using that. # Otherwise, start one. # End this watch when there are 5 healthy Pods for the StatefulSet kubectl get pods --watch -l app=nginx In another terminal window, use kubectl scale to scale the number of replicas to 5:kubectl scale sts web --replicas=5 statefulset.apps/web scaled Examine the output of the kubectl get command in the first terminal, and wait for the three additional Pods to transition to Running and Ready.# This should already be running kubectl get pod --watch -l app=nginx NAME READY STATUS RESTARTS AGE web-0 1/1 Running 0 2h web-1 1/1 Running 0 2h NAME READY STATUS RESTARTS AGE web-2 0/1 Pending 0 0s web-2 0/1 Pending 0 0s web-2 0/1 ContainerCreating 0 0s web-2 1/1 Running 0 19s web-3 0/1 Pending 0 0s web-3 0/1 Pending 0 0s web-3 0/1 ContainerCreating 0 0s web-3 1/1 Running 0 18s web-4 0/1 Pending 0 0s web-4 0/1 Pending 0 0s web-4 0/1 ContainerCreating 0 0s web-4 1/1 Running 0 19s The StatefulSet controller scaled the number of replicas. As with StatefulSet creation, the StatefulSet controller created each Pod sequentially with respect to its ordinal index, and it waited for each Pod's predecessor to be Running and Ready before launching the subsequent Pod.Scaling downScaling down means reducing the number of replicas. For example, you might do this because the level of traffic to a service has decreased, and at the current scale there are idle resources.In one terminal, watch the StatefulSet's Pods:# End this watch when there are only 3 Pods for the StatefulSet kubectl get pod --watch -l app=nginx In another terminal, use kubectl patch to scale the StatefulSet back down to three replicas:kubectl patch sts web -p '{"spec":{"replicas":3}}' statefulset.apps/web patched Wait for web-4 and web-3 to transition to Terminating.# This should already be running kubectl get pods --watch -l app=nginx NAME READY STATUS RESTARTS AGE web-0 1/1 Running 0 3h web-1 1/1 Running 0 3h web-2 1/1 Running 0 55s web-3 1/1 Running 0 36s web-4 0/1 ContainerCreating 0 18s NAME READY STATUS RESTARTS AGE web-4 1/1 Running 0 19s web-4 1/1 Terminating 0 24s web-4 1/1 Terminating 0 24s web-3 1/1 Terminating 0 42s web-3 1/1 Terminating 0 42s Ordered Pod terminationThe control plane deleted one Pod at a time, in reverse order with respect to its ordinal index, and it waited for each Pod to be completely shut down before deleting the


# Manage Kubernetes Objects

This is the multi-page printable view of this section. Click here to print.Return to the regular view of this page.Manage Kubernetes ObjectsDeclarative and imperative paradigms for interacting with the Kubernetes API.1: Declarative Management of Kubernetes Objects Using Configuration Files2: Declarative Management of Kubernetes Objects Using Kustomize3: Managing Kubernetes Objects Using Imperative Commands4: Imperative Management of Kubernetes Objects Using Configuration Files5: Update API Objects in Place Using kubectl patch6: Migrate Kubernetes Objects Using Storage Version Migration1 - Declarative Management of Kubernetes Objects Using Configuration FilesKubernetes objects can be created, updated, and deleted by storing multiple object configuration files in a directory and using kubectl apply to recursively create and update those objects as needed. This method retains writes made to live objects without merging the changes back into the object configuration files. kubectl diff also gives you a preview of what changes apply will make.Before you beginInstall kubectl.You need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. It is recommended to run this tutorial on a cluster with at least two nodes that are not acting as control plane hosts. If you do not already have a cluster, you can create one by using minikube or you can use one of these Kubernetes playgrounds:iximiuz LabsKillercodaKodeKloudPlay with KubernetesTo check the version, enter kubectl version.Trade-offsThe kubectl tool supports three kinds of object management:Imperative commandsImperative object configurationDeclarative object configurationSee Kubernetes Object Management for a discussion of the advantages and disadvantage of each kind of object management.OverviewDeclarative object configuration requires a firm understanding of the Kubernetes object definitions and configuration. Read and complete the following documents if you have not already:Managing Kubernetes Objects Using Imperative CommandsImperative Management of Kubernetes Objects Using Configuration FilesFollowing are definitions for terms used in this document:object configuration file / configuration file: A file that defines the configuration for a Kubernetes object. This topic shows how to pass configuration files to kubectl apply. Configuration files are typically stored in source control, such as Git.live object configuration / live configuration: The live configuration values of an object, as observed by the Kubernetes cluster. These are kept in the Kubernetes cluster storage, typically etcd.declarative configuration writer / declarative writer: A person or software component that makes updates to a live object. The live writers referred to in this topic make changes to object configuration files and run kubectl apply to write the changes.How to create objectsUse kubectl apply to create all objects, except those that already exist, defined by configuration files in a specified directory:kubectl apply -f <directory> This sets the kubectl.kubernetes.io/last-applied-configuration: '{...}' annotation on each object. The annotation contains the contents of the object configuration file that was used to create the object.Note:Add the -R flag to recursively process directories.Here's an example of an object configuration file:application/simple_deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment spec: selector: matchLabels: app: nginx minReadySeconds: 5 template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx:1.14.2 ports: - containerPort: 80 Run kubectl diff to print the object that will be created:kubectl diff -f https://k8s.io/examples/application/simple_deployment.yaml Note:diff uses server-side dry-run, which needs to be enabled on kube-apiserver.Since diff performs a server-side apply request in dry-run mode, it requires granting PATCH, CREATE, and UPDATE permissions. See Dry-Run Authorization for details.Create the object using kubectl apply:kubectl apply -f https://k8s.io/examples/application/simple_deployment.yaml Print the live configuration using kubectl get:kubectl get -f https://k8s.io/examples/application/simple_deployment.yaml -o yaml The output shows that the kubectl.kubernetes.io/last-applied-configuration annotation was written to the live configuration, and it matches the configuration file:kind: Deployment metadata: annotations: # ... # This is the json representation of simple_deployment.yaml # It was written by kubectl apply when the object was created kubectl.kubernetes.io/last-applied-configuration: | {"apiVersion":"apps/v1","kind":"Deployment", "metadata":{"annotations":{},"name":"nginx-deployment","namespace":"default"}, "spec":{"minReadySeconds":5,"selector":{"matchLabels":{"app":nginx}},"template":{"metadata":{"labels":{"app":"nginx"}}, "spec":{"containers":[{"image":"nginx:1.14.2","name":"nginx", "ports":[{"containerPort":80}]}]}}}} # ... spec: # ... minReadySeconds: 5 selector: matchLabels: # ... app: nginx template: metadata: # ... labels: app: nginx spec: containers: - image: nginx:1.14.2 # ... name: nginx ports: - containerPort: 80 # ... # ... # ... # ... How to update objectsYou can also use kubectl apply to update all objects defined in a directory, even if those objects already exist. This approach accomplishes the following:Sets fields that appear in the configuration file in the live configuration.Clears fields removed from the configuration file in the live configuration.kubectl diff -f <directory> kubectl apply -f <directory> Note:Add the -R flag to recursively process directories.Here's an example configuration file:application/simple_deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment spec: selector: matchLabels: app: nginx minReadySeconds: 5 template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx:1.14.2 ports: - containerPort: 80 Create the object using kubectl apply:kubectl apply -f https://k8s.io/examples/application/simple_deployment.yaml Note:For purposes of illustration, the preceding command refers to a single configuration file instead of a directory.Print the live configuration using kubectl get:kubectl get -f https://k8s.io/examples/application/simple_deployment.yaml -o yaml The output shows that the kubectl.kubernetes.io/last-applied-configuration annotation was written to the live configuration, and it matches the configuration file:kind: Deployment metadata: annotations: # ... # This is the json representation of simple_deployment.yaml # It was written by kubectl apply when the object was created kubectl.kubernetes.io/last-applied-configuration: | {"apiVersion":"apps/v1","kind":"Deployment", "metadata":{"annotations":{},"name":"nginx-deployment","namespace":"default"}, "spec":{"minReadySeconds":5,"selector":{"matchLabels":{"app":nginx}},"template":{"metadata":{"labels":{"app":"nginx"}}, "spec":{"containers":[{"image":"nginx:1.14.2","name":"nginx", "ports":[{"containerPort":80}]}]}}}} # ... spec: # ... minReadySeconds: 5 selector: matchLabels: # ... app: nginx template: metadata: # ... labels: app: nginx spec: containers: - image: nginx:1.14.2 # ... name: nginx ports: - containerPort: 80 # ... # ... # ... # ... Directly update the replicas field in the live configuration by using kubectl scale. This does not use kubectl apply:kubectl scale deployment/nginx-deployment --replicas=2 Print the live configuration using kubectl get:kubectl get deployment nginx-deployment -o yaml The output shows that the replicas field has been set to 2, and the last-applied-configuration annotation does not contain a replicas field:apiVersion: apps/v1 kind: Deployment metadata: annotations: # ... # note that the annotation does not contain replicas # because it was not updated through apply kubectl.kubernetes.io/last-applied-configuration: | {"apiVersion":"apps/v1","kind":"Deployment", "metadata":{"annotations":{},"name":"nginx-deployment","namespace":"default"}, "spec":{"minReadySeconds":5,"selector":{"matchLabels":{"app":nginx}},"template":{"metadata":{"labels":{"app":"nginx"}}, "spec":{"containers":[{"image":"nginx:1.14.2","name":"nginx", "ports":[{"containerPort":80}]}]}}}} # ... spec: replicas: 2 # written by scale # ... minReadySeconds: 5 selector: matchLabels: # ... app: nginx template: metadata: # ... labels: app: nginx spec: containers: - image: nginx:1.14.2 # ... name: nginx ports: - containerPort: 80 # ... Update the simple_deployment.yaml configuration file to change the image from nginx:1.14.2 to nginx:1.16.1, and delete the minReadySeconds field:application/update_deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment spec: selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx:1.16.1 # update the image ports: - containerPort: 80 Apply the changes made to the configuration file:kubectl diff -f https://k8s.io/examples/application/update_deployment.yaml kubectl apply -f https://k8s.io/examples/application/update_deployment.yaml Print the live configuration using kubectl get:kubectl get -f https://k8s.io/examples/application/update_deployment.yaml -o yaml The output shows the following changes to the live configuration:The replicas field retains the value of 2 set by kubectl scale. This is possible because it is omitted from the configuration file.The image field has been updated to nginx:1.16.1 from nginx:1.14.2.The last-applied-configuration annotation has been updated with the new image.The minReadySeconds field has been cleared.The last-applied-configuration annotation no longer contains the minReadySeconds field.apiVersion: apps/v1 kind: Deployment metadata: annotations: # ... # The annotation contains the updated image to nginx 1.16.1, # but does not contain the updated replicas to 2 kubectl.kubernetes.io/last-applied-configuration: | {"apiVersion":"apps/v1","kind":"Deployment", "metadata":{"annotations":{},"name":"nginx-deployment","namespace":"default"}, "spec":{"selector":{"matchLabels":{"app":nginx}},"template":{"metadata":{"labels":{"app":"nginx"}}, "spec":{"containers":[{"image":"nginx:1.16.1","name":"nginx", "ports":[{"containerPort":80}]}]}}}} # ... spec: replicas: 2 # Set by `kubectl scale`. Ignored by `kubectl apply`. # minReadySeconds cleared by `kubectl apply` # ... selector: matchLabels: # ... app: nginx template: metadata: # ... labels: app: nginx spec: containers: - image: nginx:1.16.1 # Set by `kubectl apply` # ... name: nginx ports: - containerPort: 80 # ... # ... # ... # ... Warning:Mixing kubectl apply with the imperative object configuration commands create and replace is not supported. This is because create and replace do not retain the kubectl.kubernetes.io/last-applied-configuration that kubectl apply uses to compute updates.How to delete objectsThere are two approaches to delete objects managed by kubectl apply.Recommended: kubectl delete -f <filename>Manually deleting objects using the imperative command is the recommended approach, as it is more explicit about what is being deleted, and less likely to result in the user deleting something unintentionally:kubectl delete -f <filename> Alternative: kubectl apply -f <directory> --pruneAs an alternative to kubectl delete, you can use kubectl apply to identify objects to be deleted after their manifests have been removed from a directory in the local filesystem.In Kubernetes 1.34, there are two pruning modes available in kubectl apply:Allowlist-based pruning: This mode has existed since kubectl v1.5 but is still in alpha due to usability, correctness and performance issues with its design. The ApplySet-based mode is designed to replace it.ApplySet-based pruning: An apply set is a server-side object (by default, a Secret) that kubectl can use to accurately and efficiently track set membership across apply operations. This mode was introduced in alpha in kubectl v1.27 as a replacement for allowlist-based pruning.FEATURE STATE: Kubernetes v1.5 [alpha]Warning:Take care when using --prune with kubectl apply in allow list mode. Which objects are pruned depends on the values of the --prune-allowlist, --selector and --namespace flags, and relies on dynamic discovery of the objects in scope. Especially if flag values are changed between invocations, this can lead to objects being unexpectedly deleted or retained.To use allowlist-based pruning, add the following flags to your kubectl apply invocation:--prune: Delete previously applied objects that are not in the set passed to the current invocation.--prune-allowlist: A list of group-version-kinds (GVKs) to consider for pruning. This flag is optional but strongly encouraged, as its default value is a partial list of both namespaced and cluster-scoped types, which can lead to surprising results.--selector/-l: Use a label selector to constrain the set of objects selected for pruning. This flag is optional but strongly encouraged.--all: use instead of --selector/-l to explicitly select all previously applied objects of the allowlisted types.Allowlist-based pruning queries the API server for all objects of the allowlisted GVKs that match the given labels (if any), and attempts to match the returned live object configurations against the object manifest files. If an object matches the query, and it does not have a manifest in the directory, and it has a kubectl.kubernetes.io/last-applied-configuration annotation, it is deleted.kubectl apply -f <directory> --prune -l <labels> --prune-allowlist=<gvk-list> Warning:Apply with prune should only be run against the root directory containing the object manifests. Running against sub-directories can cause objects to be unintentionally deleted if they were previously applied, have the labels given (if any), and do not appear in the subdirectory.FEATURE STATE: Kubernetes v1.27 [alpha]Caution:kubectl apply --prune --applyset is in alpha, and backwards incompatible changes might be introduced in subsequent releases.To use ApplySet-based pruning, set the KUBECTL_APPLYSET=true environment variable, and add the following flags to your kubectl apply invocation:--prune: Delete previously applied objects that are not in the set passed to the current invocation.--applyset: The name of an object that kubectl can use to accurately and efficiently track set membership across apply operations.KUBECTL_APPLYSET=true kubectl apply -f <directory> --prune --applyset=<name> By default, the type of the ApplySet parent object used is a Secret. However, ConfigMaps can also be used in the format: --applyset=configmaps/<name>. When using a Secret or ConfigMap, kubectl will create the object if it does not already exist.It is also possible to use custom resources as ApplySet parent objects. To enable this, label the Custom Resource Definition (CRD) that defines the resource you want to use with the following: applyset.kubernetes.io/is-parent-type: true. Then, create the object you want to use as an ApplySet parent (kubectl does not do this automatically for custom resources). Finally, refer to that object in the applyset flag as follows: --applyset=<resource>.<group>/<name> (for example, widgets.custom.example.com/widget-name).With ApplySet-based pruning, kubectl adds the applyset.kubernetes.io/part-of=<parentID> label to each object in the set before they are sent to the server. For performance reasons, it also collects the list of resource types and namespaces that the set contains and adds these in annotations on the live parent object. Finally, at the end of the apply operation, it queries the API server for objects of those types in those namespaces (or in the cluster scope, as applicable) that belong to the set, as defined by the applyset.kubernetes.io/part-of=<parentID> label.Caveats and restrictions:Each object may be a member of at most one set.The --namespace flag is required when using any namespaced parent, including the default Secret. This means that ApplySets spanning multiple namespaces must use a cluster-scoped custom resource as the parent object.To safely use ApplySet-based pruning with multiple directories, use a unique ApplySet name for each.How to view an objectYou can use kubectl get with -o yaml to view the configuration of a live object:kubectl get -f <filename|url> -o yaml How apply calculates differences and merges changesCaution:A patch is an update operation that is scoped to specific fields of an object instead of the entire object. This enables updating only a specific set of fields on an object without reading the object first.When kubectl apply updates the live configuration for an object, it does so by sending a patch request to the API server. The patch defines updates scoped to specific fields of the live object configuration. The kubectl apply command calculates this patch request using the configuration file, the live configuration, and the last-applied-configuration annotation stored in the live configuration.Merge patch calculationThe kubectl apply command writes the contents of the configuration file to the kubectl.kubernetes.io/last-applied-configuration annotation. This is used to identify fields that have been removed from the configuration file and need to be cleared from the live configuration. Here are the steps used to calculate which fields should be deleted or set:Calculate the fields to delete. These are the fields present in last-applied-configuration and missing from the configuration file.Calculate the fields to add or set. These are the fields present in the configuration file whose values don't match the live configuration.Here's an example. Suppose this is the configuration file for a Deployment object:application/update_deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment spec: selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx:1.16.1 # update the image ports: - containerPort: 80 Also, suppose this is the live configuration for the same Deployment object:apiVersion: apps/v1 kind: Deployment metadata: annotations: # ... # note that the annotation does not contain replicas # because it was not updated through apply kubectl.kubernetes.io/last-applied-configuration: | {"apiVersion":"apps/v1","kind":"Deployment", "metadata":{"annotations":{},"name":"nginx-deployment","namespace":"default"}, "spec":{"minReadySeconds":5,"selector":{"matchLabels":{"app":nginx}},"template":{"metadata":{"labels":{"app":"nginx"}}, "spec":{"containers":[{"image":"nginx:1.14.2","name":"nginx", "ports":[{"containerPort":80}]}]}}}} # ... spec: replicas: 2 # written by scale # ... minReadySeconds: 5 selector: matchLabels: # ... app: nginx template: metadata: # ... labels: app: nginx spec: containers: - image: nginx:1.14.2 # ... name: nginx ports: - containerPort: 80 # ... Here are the merge calculations that would be performed by kubectl apply:Calculate the fields to delete by reading values from last-applied-configuration and comparing them to values in the configuration file. Clear fields explicitly set to null in the local object configuration file regardless of whether they appear in the last-applied-configuration. In this example, minReadySeconds appears in the last-applied-configuration annotation, but does not appear in the configuration file. Action: Clear minReadySeconds from the live configuration.Calculate the fields to set by reading values from the configuration file and comparing them to values in the live configuration. In this example, the value of image in the configuration file does not match the value in the live configuration. Action: Set the value of image in the live configuration.Set the last-applied-configuration annotation to match the value of the configuration file.Merge the results from 1, 2, 3 into a single patch request to the API server.Here is the live configuration that is the result of the merge:apiVersion: apps/v1 kind: Deployment metadata: annotations: # ... # The annotation contains the updated image to nginx 1.16.1, # but does not contain the updated replicas to 2 kubectl.kubernetes.io/last-applied-configuration: | {"apiVersion":"apps/v1","kind":"Deployment", "metadata":{"annotations":{},"name":"nginx-deployment","namespace":"default"}, "spec":{"selector":{"matchLabels":{"app":nginx}},"template":{"metadata":{"labels":{"app":"nginx"}}, "spec":{"containers":[{"image":"nginx:1.16.1","name":"nginx", "ports":[{"containerPort":80}]}]}}}} # ... spec: selector: matchLabels: # ... app: nginx replicas: 2 # Set by `kubectl scale`. Ignored by `kubectl apply`. # minReadySeconds cleared by `kubectl apply` # ... template: metadata: # ... labels: app: nginx spec: containers: - image: nginx:1.16.1 # Set by `kubectl apply` # ... name: nginx ports: - containerPort: 80 # ... # ... # ... # ... How different types of fields are mergedHow a particular field in a configuration file is merged with the live configuration depends on the type of the field. There are several types of fields:primitive: A field of type string, integer, or boolean. For example, image and replicas are primitive fields. Action: Replace.map, also called object: A field of type map or a complex type that contains subfields. For example, labels, annotations,spec and metadata are all maps. Action: Merge elements or subfields.list: A field containing a list of items that can be either primitive types or maps. For example, containers, ports, and args are lists. Action: Varies.When kubectl apply updates a map or list field, it typically does not replace the entire field, but instead updates the individual subelements. For instance, when merging the spec on a Deployment, the entire spec is not replaced. Instead the subfields of spec, such as replicas, are compared and merged.Merging changes to primitive fieldsPrimitive fields are replaced or cleared.Note:- is used for "not applicable" because the value is not used.Field in object configuration fileField in live object configurationField in last-applied-configurationActionYesYes-Set live to configuration file value.YesNo-Set live to local configuration.No-YesClear from live configuration.No-NoDo nothing. Keep live value.Merging changes to map fieldsFields that represent maps are merged by comparing each of the subfields or elements of the map:Note:- is used for "not applicable" because the value is not used.Key in object configuration fileKey in live object configurationField in last-applied-configurationActionYesYes-Compare sub fields values.YesNo-Set live to local configuration.No-YesDelete from live configuration.No-NoDo nothing. Keep live value.Merging changes for fields of type listMerging changes to a list uses one of three strategies:Replace the list if all its elements are primitives.Merge individual elements in a list of complex elements.Merge a list of primitive elements.The choice of strategy is made on a per-field basis.Replace the list if all its elements are primitivesTreat the list the same as a primitive field. Replace or delete the entire list. This preserves ordering.Example: Use kubectl apply to update the args field of a Container in a Pod. This sets the value of args in the live configuration to the value in the configuration file. Any args elements that had previously been added to the live configuration are lost. The order of the args elements defined in the configuration file is retained in the live configuration.# last-applied-configuration value args: ["a", "b"] # configuration file value args: ["a", "c"] # live configuration args: ["a", "b", "d"] # result after merge args: ["a", "c"] Explanation: The merge used the configuration file value as the new list value.Merge individual elements of a list of complex elements:Treat the list as a map, and treat a specific field of each element as a key. Add, delete, or update individual elements. This does not preserve ordering.This merge strategy uses a special tag on each field called a patchMergeKey. The patchMergeKey is defined for each field in the Kubernetes source code: types.go When merging a list of maps, the field specified as the patchMergeKey for a given element is used like a map key for that element.Example: Use kubectl apply to update the containers field of a PodSpec. This merges the list as though it was a map where each element is keyed by name.# last-applied-configuration value containers: - name: nginx image: nginx:1.16 - name: nginx-helper-a # key: nginx-helper-a; will be deleted in result image: helper:1.3 - name: nginx-helper-b # key: nginx-helper-b; will be retained image: helper:1.3 # configuration file value containers: - name: nginx image: nginx:1.16 - name: nginx-helper-b image: helper:1.3 - name: nginx-helper-c # key: nginx-helper-c; will be added in result image: helper:1.3 # live configuration containers: - name: nginx image: nginx:1.16 - name: nginx-helper-a image: helper:1.3 - name: nginx-helper-b image: helper:1.3 args: ["run"] # Field will be retained - name: nginx-helper-d # key: nginx-helper-d; will be retained image: helper:1.3 # result after merge containers: - name: nginx image: nginx:1.16 # Element nginx-helper-a was deleted - name: nginx-helper-b image: helper:1.3 args: ["run"] # Field was retained - name: nginx-helper-c # Element was added image: helper:1.3 - name: nginx-helper-d # Element was ignored image: helper:1.3 Explanation:The container named "nginx-helper-a" was deleted because no container named "nginx-helper-a" appeared in the configuration file.The container named "nginx-helper-b" retained the changes to args in the live configuration. kubectl apply was able to identify that "nginx-helper-b" in the live configuration was the same "nginx-helper-b" as in the configuration file, even though their fields had different values (no args in the configuration file). This is because the patchMergeKey field value (name) was identical in both.The container named "nginx-helper-c" was added because no container with that name appeared in the live configuration, but one with that name appeared in the configuration file.The container named "nginx-helper-d" was retained because no element with that name appeared in the last-applied-configuration.Merge a list of primitive elementsAs of Kubernetes 1.5, merging lists of primitive elements is not supported.Note:Which of the above strategies is chosen for a given field is controlled by the patchStrategy tag in types.go If no patchStrategy is specified for a field of type list, then the list is replaced.Default field valuesThe API server sets certain fields to default values in the live configuration if they are not specified when the object is created.Here's a configuration file for a Deployment. The file does not specify strategy:application/simple_deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment spec: selector: matchLabels: app: nginx minReadySeconds: 5 template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx:1.14.2 ports: - containerPort: 80 Create the object using kubectl apply:kubectl apply -f https://k8s.io/examples/application/simple_deployment.yaml Print the live configuration using kubectl get:kubectl get -f https://k8s.io/examples/application/simple_deployment.yaml -o yaml The output shows that the API server set several fields to default values in the live configuration. These fields were not specified in the configuration file.apiVersion: apps/v1 kind: Deployment # ... spec: selector: matchLabels: app: nginx minReadySeconds: 5 replicas: 1 # defaulted by apiserver strategy: rollingUpdate: # defaulted by apiserver - derived from strategy.type maxSurge: 1 maxUnavailable: 1 type: RollingUpdate # defaulted by apiserver template: metadata: creationTimestamp: null labels: app: nginx spec: containers: - image: nginx:1.14.2 imagePullPolicy: IfNotPresent # defaulted by apiserver name: nginx ports: - containerPort: 80 protocol: TCP # defaulted by apiserver resources: {} # defaulted by apiserver terminationMessagePath: /dev/termination-log # defaulted by apiserver dnsPolicy: ClusterFirst # defaulted by apiserver restartPolicy: Always # defaulted by apiserver securityContext: {} # defaulted by apiserver terminationGracePeriodSeconds: 30 # defaulted by apiserver # ... In a patch request, defaulted fields are not re-defaulted unless they are explicitly cleared as part of a patch request. This can cause unexpected behavior for fields that are defaulted based on the values of other fields. When the other fields are later changed, the values defaulted from them will not be updated unless they are explicitly cleared.For this reason, it is recommended that certain fields defaulted by the server are explicitly defined in the configuration file, even if the desired values match the server defaults. This makes it easier to recognize conflicting values that will not be re-defaulted by the server.Example:# last-applied-configuration spec: template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx:1.14.2 ports: - containerPort: 80 # configuration file spec: strategy: type: Recreate # updated value template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx:1.14.2 ports: - containerPort: 80 # live configuration spec: strategy: type: RollingUpdate # defaulted value rollingUpdate: # defaulted value derived from type maxSurge : 1 maxUnavailable: 1 template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx:1.14.2 ports: - containerPort: 80 # result after merge - ERROR! spec: strategy: type: Recreate # updated value: incompatible with rollingUpdate rollingUpdate: # defaulted value: incompatible with "type: Recreate" maxSurge : 1 maxUnavailable: 1 template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx:1.14.2 ports: - containerPort: 80 Explanation:The user creates a Deployment without defining strategy.type.The server defaults strategy.type to RollingUpdate and defaults the strategy.rollingUpdate values.The user changes strategy.type to Recreate. The strategy.rollingUpdate values remain at their defaulted values, though the server expects them to be cleared. If the strategy.rollingUpdate values had been defined initially in the configuration file, it would have been more clear that they needed to be deleted.Apply fails because strategy.rollingUpdate is not cleared. The strategy.rollingupdate field cannot be defined with a strategy.type of Recreate.Recommendation: These fields should be explicitly defined in the object configuration file:Selectors and PodTemplate labels on workloads, such as Deployment, StatefulSet, Job, DaemonSet, ReplicaSet, and ReplicationControllerDeployment rollout strategyHow to clear server-defaulted fields or fields set by other writersFields that do not appear in the configuration file can be cleared by setting their values to null and then applying the configuration file. For fields defaulted by the server, this triggers re-defaulting the values.How to change ownership of a field between the configuration file and direct imperative writersThese are the only methods you should use to change an individual object field:Use kubectl apply.Write directly to the live configuration without modifying the configuration file: for example, use kubectl scale.Changing the owner from a direct imperative writer to a configuration fileAdd the field to the configuration file. For the field, discontinue direct updates to the live configuration that do not go through kubectl apply.Changing the owner from a configuration file to a direct imperative writerAs of Kubernetes 1.5, changing ownership of a field from a configuration file to an imperative writer requires manual steps:Remove the field from the configuration file.Remove the field from the kubectl.kubernetes.io/last-applied-configuration annotation on the live object.Changing management methodsKubernetes objects should be managed using only one method at a time. Switching from one method to another is possible, but is a manual process.Note:It is OK to use imperative deletion with declarative management.Migrating from imperative command management to declarative object configurationMigrating from imperative command management to declarative object configuration involves several manual steps:Export the live object to a local configuration file:kubectl get <kind>/<name> -o yaml > <kind>_<name>.yaml Manually remove the status field from the configuration file.Note:This step is optional, as kubectl apply does not update the status field even if it is present in the configuration file.Set the kubectl.kubernetes.io/last-applied-configuration annotation on the object:kubectl replace --save-config -f <kind>_<name>.yaml Change processes to use kubectl apply for managing the object exclusively.Migrating from imperative object configuration to declarative object configurationSet the kubectl.kubernetes.io/last-applied-configuration annotation on the object:kubectl replace --save-config -f <kind>_<name>.yaml Change processes to use kubectl apply for managing the object exclusively.Defining controller selectors and PodTemplate labelsWarning:Updating selectors on controllers is strongly discouraged.The recommended approach is to define a single, immutable PodTemplate label used only by the controller selector with no other semantic meaning.Example:selector: matchLabels: controller-selector: "apps/v1/deployment/nginx" template: metadata: labels: controller-selector: "apps/v1/deployment/nginx" What's


# Tasks

This is the multi-page printable view of this section. Click here to print.Return to the regular view of this page.Tasks1: Install Tools1.1: Install and Set Up kubectl on Linux1.2: Install and Set Up kubectl on macOS1.3: Install and Set Up kubectl on Windows2: Administer a Cluster2.1: Administration with kubeadm2.1.1: Adding Linux worker nodes2.1.2: Adding Windows worker nodes2.1.3: Upgrading kubeadm clusters2.1.4: Upgrading Linux nodes2.1.5: Upgrading Windows nodes2.1.6: Configuring a cgroup driver2.1.7: Certificate Management with kubeadm2.1.8: Reconfiguring a kubeadm cluster2.1.9: Changing The Kubernetes Package Repository2.2: Overprovision Node Capacity For A Cluster2.3: Migrating from dockershim2.3.1: Changing the Container Runtime on a Node from Docker Engine to containerd2.3.2: Find Out What Container Runtime is Used on a Node2.3.3: Troubleshooting CNI plugin-related errors2.3.4: Check whether dockershim removal affects you2.3.5: Migrating telemetry and security agents from dockershim2.4: Generate Certificates Manually2.5: Manage Memory, CPU, and API Resources2.5.1: Configure Default Memory Requests and Limits for a Namespace2.5.2: Configure Default CPU Requests and Limits for a Namespace2.5.3: Configure Minimum and Maximum Memory Constraints for a Namespace2.5.4: Configure Minimum and Maximum CPU Constraints for a Namespace2.5.5: Configure Memory and CPU Quotas for a Namespace2.5.6: Configure a Pod Quota for a Namespace2.6: Install a Network Policy Provider2.6.1: Use Antrea for NetworkPolicy2.6.2: Use Calico for NetworkPolicy2.6.3: Use Cilium for NetworkPolicy2.6.4: Use Kube-router for NetworkPolicy2.6.5: Romana for NetworkPolicy2.6.6: Weave Net for NetworkPolicy2.7: Access Clusters Using the Kubernetes API2.8: Advertise Extended Resources for a Node2.9: Autoscale the DNS Service in a Cluster2.10: Change the Access Mode of a PersistentVolume to ReadWriteOncePod2.11: Change the default StorageClass2.12: Switching from Polling to CRI Event-based Updates to Container Status2.13: Change the Reclaim Policy of a PersistentVolume2.14: Cloud Controller Manager Administration2.15: Configure a kubelet image credential provider2.16: Configure Quotas for API Objects2.17: Control CPU Management Policies on the Node2.18: Control Topology Management Policies on a node2.19: Customizing DNS Service2.20: Debugging DNS Resolution2.21: Declare Network Policy2.22: Developing Cloud Controller Manager2.23: Enable Or Disable A Kubernetes API2.24: Encrypting Confidential Data at Rest2.25: Decrypt Confidential Data that is Already Encrypted at Rest2.26: Guaranteed Scheduling For Critical Add-On Pods2.27: IP Masquerade Agent User Guide2.28: Limit Storage Consumption2.29: Migrate Replicated Control Plane To Use Cloud Controller Manager2.30: Operating etcd clusters for Kubernetes2.31: Reserve Compute Resources for System Daemons2.32: Running Kubernetes Node Components as a Non-root User2.33: Safely Drain a Node2.34: Securing a Cluster2.35: Set Kubelet Parameters Via A Configuration File2.36: Share a Cluster with Namespaces2.37: Upgrade A Cluster2.38: Use Cascading Deletion in a Cluster2.39: Using a KMS provider for data encryption2.40: Using CoreDNS for Service Discovery2.41: Using NodeLocal DNSCache in Kubernetes Clusters2.42: Using sysctls in a Kubernetes Cluster2.43: Utilizing the NUMA-aware Memory Manager2.44: Verify Signed Kubernetes Artifacts3: Configure Pods and Containers3.1: Assign Memory Resources to Containers and Pods3.2: Assign CPU Resources to Containers and Pods3.3: Assign Devices to Pods and Containers3.3.1: Set Up DRA in a Cluster3.3.2: Allocate Devices to Workloads with DRA3.4: Assign Pod-level CPU and memory resources3.5: Configure GMSA for Windows Pods and containers3.6: Resize CPU and Memory Resources assigned to Containers3.7: Configure RunAsUserName for Windows pods and containers3.8: Create a Windows HostProcess Pod3.9: Configure Quality of Service for Pods3.10: Assign Extended Resources to a Container3.11: Configure a Pod to Use a Volume for Storage3.12: Configure a Pod to Use a PersistentVolume for Storage3.13: Configure a Pod to Use a Projected Volume for Storage3.14: Configure a Security Context for a Pod or Container3.15: Configure Service Accounts for Pods3.16: Pull an Image from a Private Registry3.17: Configure Liveness, Readiness and Startup Probes3.18: Assign Pods to Nodes3.19: Assign Pods to Nodes using Node Affinity3.20: Configure Pod Initialization3.21: Attach Handlers to Container Lifecycle Events3.22: Configure a Pod to Use a ConfigMap3.23: Share Process Namespace between Containers in a Pod3.24: Use a User Namespace With a Pod3.25: Use an Image Volume With a Pod3.26: Create static Pods3.27: Translate a Docker Compose File to Kubernetes Resources3.28: Enforce Pod Security Standards by Configuring the Built-in Admission Controller3.29: Enforce Pod Security Standards with Namespace Labels3.30: Migrate from PodSecurityPolicy to the Built-In PodSecurity Admission Controller4: Monitoring, Logging, and Debugging4.1: Logging in Kubernetes4.2: Monitoring in Kubernetes4.3: Troubleshooting Applications4.3.1: Debug Pods4.3.2: Debug Services4.3.3: Debug a StatefulSet4.3.4: Determine the Reason for Pod Failure4.3.5: Debug Init Containers4.3.6: Debug Running Pods4.3.7: Get a Shell to a Running Container4.4: Troubleshooting Clusters4.4.1: Troubleshooting kubectl4.4.2: Resource metrics pipeline4.4.3: Tools for Monitoring Resources4.4.4: Monitor Node Health4.4.5: Debugging Kubernetes nodes with crictl4.4.6: Auditing4.4.7: Debugging Kubernetes Nodes With Kubectl4.4.8: Developing and debugging services locally using telepresence4.4.9: Windows debugging tips5: Manage Kubernetes Objects5.1: Declarative Management of Kubernetes Objects Using Configuration Files5.2: Declarative Management of Kubernetes Objects Using Kustomize5.3: Managing Kubernetes Objects Using Imperative Commands5.4: Imperative Management of Kubernetes Objects Using Configuration Files5.5: Update API Objects in Place Using kubectl patch5.6: Migrate Kubernetes Objects Using Storage Version Migration6: Managing Secrets6.1: Managing Secrets using kubectl6.2: Managing Secrets using Configuration File6.3: Managing Secrets using Kustomize7: Inject Data Into Applications7.1: Define a Command and Arguments for a Container7.2: Define Dependent Environment Variables7.3: Define Environment Variables for a Container7.4: Define Environment Variable Values Using An Init Container7.5: Expose Pod Information to Containers Through Environment Variables7.6: Expose Pod Information to Containers Through Files7.7: Distribute Credentials Securely Using Secrets8: Run Applications8.1: Run a Stateless Application Using a Deployment8.2: Run a Single-Instance Stateful Application8.3: Run a Replicated Stateful Application8.4: Scale a StatefulSet8.5: Delete a StatefulSet8.6: Force Delete StatefulSet Pods8.7: Horizontal Pod Autoscaling8.8: HorizontalPodAutoscaler Walkthrough8.9: Specifying a Disruption Budget for your Application8.10: Accessing the Kubernetes API from a Pod9: Run Jobs9.1: Running Automated Tasks with a CronJob9.2: Coarse Parallel Processing Using a Work Queue9.3: Fine Parallel Processing Using a Work Queue9.4: Indexed Job for Parallel Processing with Static Work Assignment9.5: Job with Pod-to-Pod Communication9.6: Parallel Processing using Expansions9.7: Handling retriable and non-retriable pod failures with Pod failure policy10: Access Applications in a Cluster10.1: Deploy and Access the Kubernetes Dashboard10.2: Accessing Clusters10.3: Configure Access to Multiple Clusters10.4: Use Port Forwarding to Access Applications in a Cluster10.5: Use a Service to Access an Application in a Cluster10.6: Connect a Frontend to a Backend Using Services10.7: Create an External Load Balancer10.8: List All Container Images Running in a Cluster10.9: Set up Ingress on Minikube with the NGINX Ingress Controller10.10: Communicate Between Containers in the Same Pod Using a Shared Volume10.11: Configure DNS for a Cluster10.12: Access Services Running on Clusters11: Extend Kubernetes11.1: Configure the Aggregation Layer11.2: Use Custom Resources11.2.1: Extend the Kubernetes API with CustomResourceDefinitions11.2.2: Versions in CustomResourceDefinitions11.3: Set up an Extension API Server11.4: Configure Multiple Schedulers11.5: Use an HTTP Proxy to Access the Kubernetes API11.6: Use a SOCKS5 Proxy to Access the Kubernetes API11.7: Set up Konnectivity service12: TLS12.1: Issue a Certificate for a Kubernetes API Client Using A CertificateSigningRequest12.2: Configure Certificate Rotation for the Kubelet12.3: Manage TLS Certificates in a Cluster12.4: Manual Rotation of CA Certificates13: Manage Cluster Daemons13.1: Building a Basic DaemonSet13.2: Perform a Rolling Update on a DaemonSet13.3: Perform a Rollback on a DaemonSet13.4: Running Pods on Only Some Nodes14: Networking14.1: Adding entries to Pod /etc/hosts with HostAliases14.2: Extend Service IP Ranges14.3: Kubernetes Default Service CIDR Reconfiguration14.4: Validate IPv4/IPv6 dual-stack15: Extend kubectl with plugins16: Manage HugePages17: Schedule GPUsThis section of the Kubernetes documentation contains pages that show how to do individual tasks. A task page shows how to do a single thing, typically by giving a short sequence of steps.If you would like to write a task page, see Creating a Documentation Pull Request.1 - Install ToolsSet up Kubernetes tools on your computer.kubectlThe Kubernetes command-line tool, kubectl, allows you to run commands against Kubernetes clusters. You can use kubectl to deploy applications, inspect and manage cluster resources, and view logs. For more information including a complete list of kubectl operations, see the kubectl reference documentation.kubectl is installable on a variety of Linux platforms, macOS and Windows. Find your preferred operating system below.Install kubectl on LinuxInstall kubectl on macOSInstall kubectl on Windowskindkind lets you run Kubernetes on your local computer. This tool requires that you have either Docker or Podman installed.The kind Quick Start page shows you what you need to do to get up and running with kind.View kind Quick Start GuideminikubeLike kind, minikube is a tool that lets you run Kubernetes locally. minikube runs an all-in-one or a multi-node local Kubernetes cluster on your personal computer (including Windows, macOS and Linux PCs) so that you can try out Kubernetes, or for daily development work.You can follow the official Get Started! guide if your focus is on getting the tool installed.View minikube Get Started! GuideOnce you have minikube working, you can use it to run a sample application.kubeadmYou can use the kubeadm tool to create and manage Kubernetes clusters. It performs the actions necessary to get a minimum viable, secure cluster up and running in a user friendly way.Installing kubeadm shows you how to install kubeadm. Once installed, you can use it to create a cluster.View kubeadm Install Guide1.1 - Install and Set Up kubectl on LinuxBefore you beginYou must use a kubectl version that is within one minor version difference of your cluster. For example, a v1.34 client can communicate with v1.33, v1.34, and v1.35 control planes. Using the latest compatible version of kubectl helps avoid unforeseen issues.Install kubectl on LinuxThe following methods exist for installing kubectl on Linux:Install kubectl binary with curl on LinuxInstall using native package managementInstall using other package managementInstall kubectl binary with curl on LinuxDownload the latest release with the command: curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl" curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/arm64/kubectl" Note:To download a specific version, replace the $(curl -L -s https://dl.k8s.io/release/stable.txt) portion of the command with the specific version.For example, to download version 1.34.0 on Linux x86-64, type:curl -LO https://dl.k8s.io/release/v1.34.0/bin/linux/amd64/kubectl And for Linux ARM64, type:curl -LO https://dl.k8s.io/release/v1.34.0/bin/linux/arm64/kubectl Validate the binary (optional)Download the kubectl checksum file: curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl.sha256" curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/arm64/kubectl.sha256" Validate the kubectl binary against the checksum file:echo "$(cat kubectl.sha256) kubectl" | sha256sum --check If valid, the output is:kubectl: OK If the check fails, sha256 exits with nonzero status and prints output similar to:kubectl: FAILED sha256sum: WARNING: 1 computed checksum did NOT match Note:Download the same version of the binary and checksum.Install kubectlsudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl Note:If you do not have root access on the target system, you can still install kubectl to the ~/.local/bin directory:chmod +x kubectl mkdir -p ~/.local/bin mv ./kubectl ~/.local/bin/kubectl # and then append (or prepend) ~/.local/bin to $PATH Test to ensure the version you installed is up-to-date:kubectl version --client Or use this for detailed view of version:kubectl version --client --output=yaml Install using native package managementUpdate the apt package index and install packages needed to use the Kubernetes apt repository:sudo apt-get update # apt-transport-https may be a dummy package; if so, you can skip that package sudo apt-get install -y apt-transport-https ca-certificates curl gnupg Download the public signing key for the Kubernetes package repositories. The same signing key is used for all repositories so you can disregard the version in the URL:# If the folder `/etc/apt/keyrings` does not exist, it should be created before the curl command, read the note below. # sudo mkdir -p -m 755 /etc/apt/keyrings curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.34/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg sudo chmod 644 /etc/apt/keyrings/kubernetes-apt-keyring.gpg # allow unprivileged APT programs to read this keyring Note:In releases older than Debian 12 and Ubuntu 22.04, folder /etc/apt/keyrings does not exist by default, and it should be created before the curl command.Add the appropriate Kubernetes apt repository. If you want to use Kubernetes version different than v1.34, replace v1.34 with the desired minor version in the command below:# This overwrites any existing configuration in /etc/apt/sources.list.d/kubernetes.list echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.34/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list sudo chmod 644 /etc/apt/sources.list.d/kubernetes.list # helps tools such as command-not-found to work correctly Note:To upgrade kubectl to another minor release, you'll need to bump the version in /etc/apt/sources.list.d/kubernetes.list before running apt-get update and apt-get upgrade. This procedure is described in more detail in Changing The Kubernetes Package Repository.Update apt package index, then install kubectl:sudo apt-get update sudo apt-get install -y kubectl Add the Kubernetes yum repository. If you want to use Kubernetes version different than v1.34, replace v1.34 with the desired minor version in the command below.# This overwrites any existing configuration in /etc/yum.repos.d/kubernetes.repo cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo [kubernetes] name=Kubernetes baseurl=https://pkgs.k8s.io/core:/stable:/v1.34/rpm/ enabled=1 gpgcheck=1 gpgkey=https://pkgs.k8s.io/core:/stable:/v1.34/rpm/repodata/repomd.xml.key EOF Note:To upgrade kubectl to another minor release, you'll need to bump the version in /etc/yum.repos.d/kubernetes.repo before running yum update. This procedure is described in more detail in Changing The Kubernetes Package Repository.Install kubectl using yum:sudo yum install -y kubectl Add the Kubernetes zypper repository. If you want to use Kubernetes version different than v1.34, replace v1.34 with the desired minor version in the command below.# This overwrites any existing configuration in /etc/zypp/repos.d/kubernetes.repo cat <<EOF | sudo tee /etc/zypp/repos.d/kubernetes.repo [kubernetes] name=Kubernetes baseurl=https://pkgs.k8s.io/core:/stable:/v1.34/rpm/ enabled=1 gpgcheck=1 gpgkey=https://pkgs.k8s.io/core:/stable:/v1.34/rpm/repodata/repomd.xml.key EOF Note:To upgrade kubectl to another minor release, you'll need to bump the version in /etc/zypp/repos.d/kubernetes.repo before running zypper update. This procedure is described in more detail in Changing The Kubernetes Package Repository.Update zypper and confirm the new repo addition:sudo zypper update When this message appears, press 't' or 'a':New repository or package signing key received: Repository: Kubernetes Key Fingerprint: 1111 2222 3333 4444 5555 6666 7777 8888 9999 AAAA Key Name: isv:kubernetes OBS Project <isv:kubernetes@build.opensuse.org> Key Algorithm: RSA 2048 Key Created: Thu 25 Aug 2022 01:21:11 PM -03 Key Expires: Sat 02 Nov 2024 01:21:11 PM -03 (expires in 85 days) Rpm Name: gpg-pubkey-9a296436-6307a177 Note: Signing data enables the recipient to verify that no modifications occurred after the data were signed. Accepting data with no, wrong or unknown signature can lead to a corrupted system and in extreme cases even to a system compromise. Note: A GPG pubkey is clearly identified by its fingerprint. Do not rely on the key's name. If you are not sure whether the presented key is authentic, ask the repository provider or check their web site. Many providers maintain a web page showing the fingerprints of the GPG keys they are using. Do you want to reject the key, trust temporarily, or trust always? [r/t/a/?] (r): a Install kubectl using zypper:sudo zypper install -y kubectl Install using other package managementIf you are on Ubuntu or another Linux distribution that supports the snap package manager, kubectl is available as a snap application.snap install kubectl --classic kubectl version --client If you are on Linux and using Homebrew package manager, kubectl is available for installation.brew install kubectl kubectl version --client Verify kubectl configurationIn order for kubectl to find and access a Kubernetes cluster, it needs a kubeconfig file, which is created automatically when you create a cluster using kube-up.sh or successfully deploy a Minikube cluster. By default, kubectl configuration is located at ~/.kube/config.Check that kubectl is properly configured by getting the cluster state:kubectl cluster-info If you see a URL response, kubectl is correctly configured to access your cluster.If you see a message similar to the following, kubectl is not configured correctly or is not able to connect to a Kubernetes cluster.The connection to the server <server-name:port> was refused - did you specify the right host or port? For example, if you are intending to run a Kubernetes cluster on your laptop (locally), you will need a tool like Minikube to be installed first and then re-run the commands stated above.If kubectl cluster-info returns the url response but you can't access your cluster, to check whether it is configured properly, use:kubectl cluster-info dump Troubleshooting the 'No Auth Provider Found' error messageIn Kubernetes 1.26, kubectl removed the built-in authentication for the following cloud providers' managed Kubernetes offerings. These providers have released kubectl plugins to provide the cloud-specific authentication. For instructions, refer to the following provider documentation:Azure AKS: kubelogin pluginGoogle Kubernetes Engine: gke-gcloud-auth-plugin(There could also be other reasons to see the same error message, unrelated to that change.)Optional kubectl configurations and pluginsEnable shell autocompletionkubectl provides autocompletion support for Bash, Zsh, Fish, and PowerShell, which can save you a lot of typing.Below are the procedures to set up autocompletion for Bash, Fish, and Zsh.IntroductionThe kubectl completion script for Bash can be generated with the command kubectl completion bash. Sourcing the completion script in your shell enables kubectl autocompletion.However, the completion script depends on bash-completion, which means that you have to install this software first (you can test if you have bash-completion already installed by running type _init_completion).Install bash-completionbash-completion is provided by many package managers (see here). You can install it with apt-get install bash-completion or yum install bash-completion, etc.The above commands create /usr/share/bash-completion/bash_completion, which is the main script of bash-completion. Depending on your package manager, you have to manually source this file in your ~/.bashrc file.To find out, reload your shell and run type _init_completion. If the command succeeds, you're already set, otherwise add the following to your ~/.bashrc file:source /usr/share/bash-completion/bash_completion Reload your shell and verify that bash-completion is correctly installed by typing type _init_completion.Enable kubectl autocompletionBashYou now need to ensure that the kubectl completion script gets sourced in all your shell sessions. There are two ways in which you can do this: echo 'source <(kubectl completion bash)' >>~/.bashrc kubectl completion bash | sudo tee /etc/bash_completion.d/kubectl > /dev/null sudo chmod a+r /etc/bash_completion.d/kubectl If you have an alias for kubectl, you can extend shell completion to work with that alias:echo 'alias k=kubectl' >>~/.bashrc echo 'complete -o default -F __start_kubectl k' >>~/.bashrc Note:bash-completion sources all completion scripts in /etc/bash_completion.d.Both approaches are equivalent. After reloading your shell, kubectl autocompletion should be working. To enable bash autocompletion in current session of shell, source the ~/.bashrc file:source ~/.bashrc Note:Autocomplete for Fish requires kubectl 1.23 or later.The kubectl completion script for Fish can be generated with the command kubectl completion fish. Sourcing the completion script in your shell enables kubectl autocompletion.To do so in all your shell sessions, add the following line to your ~/.config/fish/config.fish file:kubectl completion fish | source After reloading your shell, kubectl autocompletion should be working.The kubectl completion script for Zsh can be generated with the command kubectl completion zsh. Sourcing the completion script in your shell enables kubectl autocompletion.To do so in all your shell sessions, add the following to your ~/.zshrc file:source <(kubectl completion zsh) If you have an alias for kubectl, kubectl autocompletion will automatically work with it.After reloading your shell, kubectl autocompletion should be working.If you get an error like 2: command not found: compdef, then add the following to the beginning of your ~/.zshrc file:autoload -Uz compinit compinit Configure kubercSee kuberc for more information.Install kubectl convert pluginA plugin for Kubernetes command-line tool kubectl, which allows you to convert manifests between different API versions. This can be particularly helpful to migrate manifests to a non-deprecated api version with newer Kubernetes release. For more info, visit migrate to non deprecated apisDownload the latest release with the command: curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl-convert" curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/arm64/kubectl-convert" Validate the binary (optional)Download the kubectl-convert checksum file: curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl-convert.sha256" curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/arm64/kubectl-convert.sha256" Validate the kubectl-convert binary against the checksum file:echo "$(cat kubectl-convert.sha256) kubectl-convert" | sha256sum --check If valid, the output is:kubectl-convert: OK If the check fails, sha256 exits with nonzero status and prints output similar to:kubectl-convert: FAILED sha256sum: WARNING: 1 computed checksum did NOT match Note:Download the same version of the binary and checksum.Install kubectl-convertsudo install -o root -g root -m 0755 kubectl-convert /usr/local/bin/kubectl-convert Verify plugin is successfully installedkubectl convert --help If you do not see an error, it means the plugin is successfully installed.After installing the plugin, clean up the installation files:rm kubectl-convert kubectl-convert.sha256 What's