NamespacesIn Kubernetes, namespaces provide a mechanism for isolating groups of resources within a single cluster. Names of resources need to be unique within a namespace, but not across namespaces. Namespace-based scoping is applicable only for namespaced objects (e.g. Deployments, Services, etc.) and not for cluster-wide objects (e.g. StorageClass, Nodes, PersistentVolumes, etc.).When to Use Multiple NamespacesNamespaces are intended for use in environments with many users spread across multiple teams, or projects. For clusters with a few to tens of users, you should not need to create or think about namespaces at all. Start using namespaces when you need the features they provide.Namespaces provide a scope for names. Names of resources need to be unique within a namespace, but not across namespaces. Namespaces cannot be nested inside one another and each Kubernetes resource can only be in one namespace.Namespaces are a way to divide cluster resources between multiple users (via resource quota).It is not necessary to use multiple namespaces to separate slightly different resources, such as different versions of the same software: use labels to distinguish resources within the same namespace.Note:For a production cluster, consider not using the default namespace. Instead, make other namespaces and use those.Initial namespacesKubernetes starts with four initial namespaces:defaultKubernetes includes this namespace so that you can start using your new cluster without first creating a namespace.kube-node-leaseThis namespace holds Lease objects associated with each node. Node leases allow the kubelet to send heartbeats so that the control plane can detect node failure.kube-publicThis namespace is readable by all clients (including those not authenticated). This namespace is mostly reserved for cluster usage, in case that some resources should be visible and readable publicly throughout the whole cluster. The public aspect of this namespace is only a convention, not a requirement.kube-systemThe namespace for objects created by the Kubernetes system.Working with NamespacesCreation and deletion of namespaces are described in the Admin Guide documentation for namespaces.Note:Avoid creating namespaces with the prefix kube-, since it is reserved for Kubernetes system namespaces.Viewing namespacesYou can list the current namespaces in a cluster using:kubectl get namespace NAME STATUS AGE default Active 1d kube-node-lease Active 1d kube-public Active 1d kube-system Active 1d Setting the namespace for a requestTo set the namespace for a current request, use the --namespace flag.For example:kubectl run nginx --image=nginx --namespace=<insert-namespace-name-here> kubectl get pods --namespace=<insert-namespace-name-here> Setting the namespace preferenceYou can permanently save the namespace for all subsequent kubectl commands in that context.kubectl config set-context --current --namespace=<insert-namespace-name-here> # Validate it kubectl config view --minify | grep namespace: Namespaces and DNSWhen you create a Service, it creates a corresponding DNS entry. This entry is of the form <service-name>.<namespace-name>.svc.cluster.local, which means that if a container only uses <service-name>, it will resolve to the service which is local to a namespace. This is useful for using the same configuration across multiple namespaces such as Development, Staging and Production. If you want to reach across namespaces, you need to use the fully qualified domain name (FQDN).As a result, all namespace names must be valid RFC 1123 DNS labels.Warning:By creating namespaces with the same name as public top-level domains, Services in these namespaces can have short DNS names that overlap with public DNS records. Workloads from any namespace performing a DNS lookup without a trailing dot will be redirected to those services, taking precedence over public DNS.To mitigate this, limit privileges for creating namespaces to trusted users. If required, you could additionally configure third-party security controls, such as admission webhooks, to block creating any namespace with the name of public TLDs.Not all objects are in a namespaceMost Kubernetes resources (e.g. pods, services, replication controllers, and others) are in some namespaces. However namespace resources are not themselves in a namespace. And low-level resources, such as nodes and persistentVolumes, are not in any namespace.To see which Kubernetes resources are and aren't in a namespace:# In a namespace kubectl api-resources --namespaced=true # Not in a namespace kubectl api-resources --namespaced=false Automatic labellingFEATURE STATE: Kubernetes 1.22 [stable]The Kubernetes control plane sets an immutable label kubernetes.io/metadata.name on all namespaces. The value of the label is the namespace name.What's nextLearn more about creating a new namespace.Learn more about deleting a namespace.

Namespaces


Configure a Pod to Use a Volume for StorageThis page shows how to configure a Pod to use a Volume for storage.A Container's file system lives only as long as the Container does. So when a Container terminates and restarts, filesystem changes are lost. For more consistent storage that is independent of the Container, you can use a Volume. This is especially important for stateful applications, such as key-value stores (such as Redis) and databases.Before you beginYou need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. It is recommended to run this tutorial on a cluster with at least two nodes that are not acting as control plane hosts. If you do not already have a cluster, you can create one by using minikube or you can use one of these Kubernetes playgrounds:iximiuz LabsKillercodaKodeKloudPlay with KubernetesTo check the version, enter kubectl version.Configure a volume for a PodIn this exercise, you create a Pod that runs one Container. This Pod has a Volume of type emptyDir that lasts for the life of the Pod, even if the Container terminates and restarts. Here is the configuration file for the Pod:pods/storage/redis.yaml apiVersion: v1 kind: Pod metadata: name: redis spec: containers: - name: redis image: redis volumeMounts: - name: redis-storage mountPath: /data/redis volumes: - name: redis-storage emptyDir: {} Create the Pod:kubectl apply -f https://k8s.io/examples/pods/storage/redis.yaml Verify that the Pod's Container is running, and then watch for changes to the Pod:kubectl get pod redis --watch The output looks like this:NAME READY STATUS RESTARTS AGE redis 1/1 Running 0 13s In another terminal, get a shell to the running Container:kubectl exec -it redis -- /bin/bash In your shell, go to /data/redis, and then create a file:root@redis:/data# cd /data/redis/ root@redis:/data/redis# echo Hello > test-file In your shell, list the running processes:root@redis:/data/redis# apt-get update root@redis:/data/redis# apt-get install procps root@redis:/data/redis# ps aux The output is similar to this:USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND redis 1 0.1 0.1 33308 3828 ? Ssl 00:46 0:00 redis-server *:6379 root 12 0.0 0.0 20228 3020 ? Ss 00:47 0:00 /bin/bash root 15 0.0 0.0 17500 2072 ? R+ 00:48 0:00 ps aux In your shell, kill the Redis process:root@redis:/data/redis# kill <pid> where <pid> is the Redis process ID (PID).In your original terminal, watch for changes to the Redis Pod. Eventually, you will see something like this:NAME READY STATUS RESTARTS AGE redis 1/1 Running 0 13s redis 0/1 Completed 0 6m redis 1/1 Running 1 6m At this point, the Container has terminated and restarted. This is because the Redis Pod has a restartPolicy of Always.Get a shell into the restarted Container:kubectl exec -it redis -- /bin/bash In your shell, go to /data/redis, and verify that test-file is still there.root@redis:/data/redis# cd /data/redis/ root@redis:/data/redis# ls test-file Delete the Pod that you created for this exercise:kubectl delete pod redis What's nextSee Volume.See Pod.In addition to the local disk storage provided by emptyDir, Kubernetes supports many different network-attached storage solutions, including PD on GCE and EBS on EC2, which are preferred for critical data and will handle details such as mounting and unmounting the devices on the nodes. See Volumes for more details.


Developing and debugging services locally using telepresenceNote: This section links to third party projects that provide functionality required by Kubernetes. The Kubernetes project authors aren't responsible for these projects, which are listed alphabetically. To add a project to this list, read the content guide before submitting a change. More information.Kubernetes applications usually consist of multiple, separate services, each running in its own container. Developing and debugging these services on a remote Kubernetes cluster can be cumbersome, requiring you to get a shell on a running container in order to run debugging tools.telepresence is a tool to ease the process of developing and debugging services locally while proxying the service to a remote Kubernetes cluster. Using telepresence allows you to use custom tools, such as a debugger and IDE, for a local service and provides the service full access to ConfigMap, secrets, and the services running on the remote cluster.This document describes using telepresence to develop and debug services running on a remote cluster locally.Before you beginKubernetes cluster is installedkubectl is configured to communicate with the clusterTelepresence is installedConnecting your local machine to a remote Kubernetes clusterAfter installing telepresence, run telepresence connect to launch its Daemon and connect your local workstation to the cluster.$ telepresence connect Launching Telepresence Daemon ... Connected to context default (https://<cluster public IP>) You can curl services using the Kubernetes syntax e.g. curl -ik https://kubernetes.defaultDeveloping or debugging an existing serviceWhen developing an application on Kubernetes, you typically program or debug a single service. The service might require access to other services for testing and debugging. One option is to use the continuous deployment pipeline, but even the fastest deployment pipeline introduces a delay in the program or debug cycle.Use the telepresence intercept $SERVICE_NAME --port $LOCAL_PORT:$REMOTE_PORT command to create an "intercept" for rerouting remote service traffic.Where:$SERVICE_NAME is the name of your local service$LOCAL_PORT is the port that your service is running on your local workstationAnd $REMOTE_PORT is the port your service listens to in the clusterRunning this command tells Telepresence to send remote traffic to your local service instead of the service in the remote Kubernetes cluster. Make edits to your service source code locally, save, and see the corresponding changes when accessing your remote application take effect immediately. You can also run your local service using a debugger or any other local development tool.How does Telepresence work?Telepresence installs a traffic-agent sidecar next to your existing application's container running in the remote cluster. It then captures all traffic requests going into the Pod, and instead of forwarding this to the application in the remote cluster, it routes all traffic (when you create a global intercept or a subset of the traffic (when you create a personal intercept) to your local development environment.What's nextIf you're interested in a hands-on tutorial, check out this tutorial that walks through locally developing the Guestbook application on Google Kubernetes Engine.For further reading, visit the Telepresence website.Items on this page refer to third party products or projects that provide functionality required by Kubernetes. The Kubernetes project authors aren't responsible for those third-party products or projects. See the CNCF website guidelines for more details.You should read the content guide before proposing a change that adds an extra third-party link.


ObjectMetaObjectMeta is metadata that all persisted resources must have, which includes all objects users must create.import "k8s.io/apimachinery/pkg/apis/meta/v1"ObjectMeta is metadata that all persisted resources must have, which includes all objects users must create.name (string)Name must be unique within a namespace. Is required when creating resources, although some resources may allow a client to request the generation of an appropriate name automatically. Name is primarily intended for creation idempotence and configuration definition. Cannot be updated. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names#namesgenerateName (string)GenerateName is an optional prefix, used by the server, to generate a unique name ONLY IF the Name field has not been provided. If this field is used, the name returned to the client will be different than the name passed. This value will also be combined with a unique suffix. The provided value has the same validation rules as the Name field, and may be truncated by the length of the suffix required to make the value unique on the server.If this field is specified and the generated name exists, the server will return a 409.Applied only if Name is not specified. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#idempotencynamespace (string)Namespace defines the space within which each name must be unique. An empty namespace is equivalent to the "default" namespace, but "default" is the canonical representation. Not all objects are required to be scoped to a namespace - the value of this field for those objects will be empty.Must be a DNS_LABEL. Cannot be updated. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaceslabels (map[string]string)Map of string keys and values that can be used to organize and categorize (scope and select) objects. May match selectors of replication controllers and services. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/labelsannotations (map[string]string)Annotations is an unstructured key value map stored with a resource that may be set by external tools to store and retrieve arbitrary metadata. They are not queryable and should be preserved when modifying objects. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/annotationsSystemfinalizers ([]string)Set: unique values will be kept during a mergeMust be empty before the object is deleted from the registry. Each entry is an identifier for the responsible component that will remove the entry from the list. If the deletionTimestamp of the object is non-nil, entries in this list can only be removed. Finalizers may be processed and removed in any order. Order is NOT enforced because it introduces significant risk of stuck finalizers. finalizers is a shared field, any actor with permission can reorder it. If the finalizer list is processed in order, then this can lead to a situation in which the component responsible for the first finalizer in the list is waiting for a signal (field value, external system, or other) produced by a component responsible for a finalizer later in the list, resulting in a deadlock. Without enforced ordering finalizers are free to order amongst themselves and are not vulnerable to ordering changes in the list.managedFields ([]ManagedFieldsEntry)Atomic: will be replaced during a mergeManagedFields maps workflow-id and version to the set of fields that are managed by that workflow. This is mostly for internal housekeeping, and users typically shouldn't need to set or understand this field. A workflow can be the user's name, a controller's name, or the name of a specific apply path like "ci-cd". The set of fields is always in the version that the workflow used when modifying the object.ManagedFieldsEntry is a workflow-id, a FieldSet and the group version of the resource that the fieldset applies to.managedFields.apiVersion (string)APIVersion defines the version of this resource that this field set applies to. The format is "group/version" just like the top-level APIVersion field. It is necessary to track the version of a field set because it cannot be automatically converted.managedFields.fieldsType (string)FieldsType is the discriminator for the different fields format and version. There is currently only one possible value: "FieldsV1"managedFields.fieldsV1 (FieldsV1)FieldsV1 holds the first JSON version format as described in the "FieldsV1" type.*FieldsV1 stores a set of fields in a data structure like a Trie, in JSON format.Each key is either a '.' representing the field itself, and will always map to an empty set, or a string representing a sub-field or item. The string will follow one of these four formats: 'f:', where is the name of a field in a struct, or key in a map 'v:', where is the exact json formatted value of a list item 'i:', where is position of a item in a list 'k:', where is a map of a list item's key fields to their unique values If a key maps to an empty Fields value, the field that key represents is part of the set.The exact format is defined in sigs.k8s.io/structured-merge-diff*managedFields.manager (string)Manager is an identifier of the workflow managing these fields.managedFields.operation (string)Operation is the type of operation which lead to this ManagedFieldsEntry being created. The only valid values for this field are 'Apply' and 'Update'.managedFields.subresource (string)Subresource is the name of the subresource used to update that object, or empty string if the object was updated through the main resource. The value of this field is used to distinguish between managers, even if they share the same name. For example, a status update will be distinct from a regular update using the same manager name. Note that the APIVersion field is not related to the Subresource field and it always corresponds to the version of the main resource.managedFields.time (Time)Time is the timestamp of when the ManagedFields entry was added. The timestamp will also be updated if a field is added, the manager changes any of the owned fields value or removes a field. The timestamp does not update when a field is removed from the entry because another manager took it over.Time is a wrapper around time.Time which supports correct marshaling to YAML and JSON. Wrappers are provided for many of the factory methods that the time package offers.ownerReferences ([]OwnerReference)Patch strategy: merge on key uidMap: unique values on key uid will be kept during a mergeList of objects depended by this object. If ALL objects in the list have been deleted, this object will be garbage collected. If this object is managed by a controller, then an entry in this list will point to this controller, with the controller field set to true. There cannot be more than one managing controller.OwnerReference contains enough information to let you identify an owning object. An owning object must be in the same namespace as the dependent, or be cluster-scoped, so there is no namespace field.ownerReferences.apiVersion (string), requiredAPI version of the referent.ownerReferences.kind (string), requiredKind of the referent. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kindsownerReferences.name (string), requiredName of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names#namesownerReferences.uid (string), requiredUID of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names#uidsownerReferences.blockOwnerDeletion (boolean)If true, AND if the owner has the "foregroundDeletion" finalizer, then the owner cannot be deleted from the key-value store until this reference is removed. See https://kubernetes.io/docs/concepts/architecture/garbage-collection/#foreground-deletion for how the garbage collector interacts with this field and enforces the foreground deletion. Defaults to false. To set this field, a user needs "delete" permission of the owner, otherwise 422 (Unprocessable Entity) will be returned.ownerReferences.controller (boolean)If true, this reference points to the managing controller.Read-onlycreationTimestamp (Time)CreationTimestamp is a timestamp representing the server time when this object was created. It is not guaranteed to be set in happens-before order across separate operations. Clients may not set this value. It is represented in RFC3339 form and is in UTC.Populated by the system. Read-only. Null for lists. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadataTime is a wrapper around time.Time which supports correct marshaling to YAML and JSON. Wrappers are provided for many of the factory methods that the time package offers.deletionGracePeriodSeconds (int64)Number of seconds allowed for this object to gracefully terminate before it will be removed from the system. Only set when deletionTimestamp is also set. May only be shortened. Read-only.deletionTimestamp (Time)DeletionTimestamp is RFC 3339 date and time at which this resource will be deleted. This field is set by the server when a graceful deletion is requested by the user, and is not directly settable by a client. The resource is expected to be deleted (no longer visible from resource lists, and not reachable by name) after the time in this field, once the finalizers list is empty. As long as the finalizers list contains items, deletion is blocked. Once the deletionTimestamp is set, this value may not be unset or be set further into the future, although it may be shortened or the resource may be deleted prior to this time. For example, a user may request that a pod is deleted in 30 seconds. The Kubelet will react by sending a graceful termination signal to the containers in the pod. After that 30 seconds, the Kubelet will send a hard termination signal (SIGKILL) to the container and after cleanup, remove the pod from the API. In the presence of network partitions, this object may still exist after this timestamp, until an administrator or automated process can determine the resource is fully terminated. If not set, graceful deletion of the object has not been requested.Populated by the system when a graceful deletion is requested. Read-only. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadataTime is a wrapper around time.Time which supports correct marshaling to YAML and JSON. Wrappers are provided for many of the factory methods that the time package offers.generation (int64)A sequence number representing a specific generation of the desired state. Populated by the system. Read-only.resourceVersion (string)An opaque value that represents the internal version of this object that can be used by clients to determine when objects have changed. May be used for optimistic concurrency, change detection, and the watch operation on a resource or set of resources. Clients must treat these values as opaque and passed unmodified back to the server. They may only be valid for a particular resource or set of resources.Populated by the system. Read-only. Value must be treated as opaque by clients and . More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#concurrency-control-and-consistencyselfLink (string)Deprecated: selfLink is a legacy read-only field that is no longer populated by the system.uid (string)UID is the unique in time and space value for this object. It is typically generated by the server on successful creation of a resource and is not allowed to change on PUT operations.Populated by the system. Read-only. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names#uidsThis page is automatically generated.If you plan to report an issue with this page, mention that the page is auto-generated in your issue description. The fix may need to happen elsewhere in the Kubernetes project.

ObjectMeta


kubeadm upgrade phaseskubeadm upgrade apply phaseUsing the phases of kubeadm upgrade apply, you can choose to execute the separate steps of the initial upgrade of a control plane node.SynopsisUse this command to invoke single phase of the "apply" workflowkubeadm upgrade apply phase [flags] Options-h, --helphelp for phaseOptions inherited from parent commands--rootfs stringThe path to the 'real' host root filesystem. This will cause kubeadm to chroot into the provided path.SynopsisRun preflight checks before upgradekubeadm upgrade apply phase preflight [flags] Options--allow-experimental-upgradesShow unstable versions of Kubernetes as an upgrade alternative and allow upgrading to an alpha/beta/release candidate versions of Kubernetes.--allow-release-candidate-upgradesShow release candidate versions of Kubernetes as an upgrade alternative and allow upgrading to a release candidate versions of Kubernetes.--config stringPath to a kubeadm configuration file.--dry-runDo not change any state, just output what actions would be performed.-f, --forceForce upgrading although some requirements might not be met. This also implies non-interactive mode.-h, --helphelp for preflight--ignore-preflight-errors stringsA list of checks whose errors will be shown as warnings. Example: 'IsPrivilegedUser,Swap'. Value 'all' ignores errors from all checks.--kubeconfig string Default: "/etc/kubernetes/admin.conf"The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations can be searched for an existing kubeconfig file.-y, --yesPerform the upgrade and do not prompt for confirmation (non-interactive mode).Options inherited from parent commands--rootfs stringThe path to the 'real' host root filesystem. This will cause kubeadm to chroot into the provided path.SynopsisUpgrade the control planekubeadm upgrade apply phase control-plane [flags] Options--certificate-renewal Default: truePerform the renewal of certificates used by component changed during upgrades.--config stringPath to a kubeadm configuration file.--dry-runDo not change any state, just output what actions would be performed.--etcd-upgrade Default: truePerform the upgrade of etcd.-h, --helphelp for control-plane--kubeconfig string Default: "/etc/kubernetes/admin.conf"The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations can be searched for an existing kubeconfig file.--patches stringPath to a directory that contains files named "target[suffix][+patchtype].extension". For example, "kube-apiserver0+merge.yaml" or just "etcd.json". "target" can be one of "kube-apiserver", "kube-controller-manager", "kube-scheduler", "etcd", "kubeletconfiguration", "corednsdeployment". "patchtype" can be one of "strategic", "merge" or "json" and they match the patch formats supported by kubectl. The default "patchtype" is "strategic". "extension" must be either "json" or "yaml". "suffix" is an optional string that can be used to determine which patches are applied first alpha-numerically.Options inherited from parent commands--rootfs stringThe path to the 'real' host root filesystem. This will cause kubeadm to chroot into the provided path.SynopsisUpload the kubeadm and kubelet configurations to ConfigMapskubeadm upgrade apply phase upload-config [flags] Options-h, --helphelp for upload-configOptions inherited from parent commands--rootfs stringThe path to the 'real' host root filesystem. This will cause kubeadm to chroot into the provided path.SynopsisUpgrade the kubelet configuration for this node by downloading it from the kubelet-config ConfigMap stored in the clusterkubeadm upgrade apply phase kubelet-config [flags] Options--config stringPath to a kubeadm configuration file.--dry-runDo not change any state, just output what actions would be performed.-h, --helphelp for kubelet-config--kubeconfig string Default: "/etc/kubernetes/admin.conf"The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations can be searched for an existing kubeconfig file.--patches stringPath to a directory that contains files named "target[suffix][+patchtype].extension". For example, "kube-apiserver0+merge.yaml" or just "etcd.json". "target" can be one of "kube-apiserver", "kube-controller-manager", "kube-scheduler", "etcd", "kubeletconfiguration", "corednsdeployment". "patchtype" can be one of "strategic", "merge" or "json" and they match the patch formats supported by kubectl. The default "patchtype" is "strategic". "extension" must be either "json" or "yaml". "suffix" is an optional string that can be used to determine which patches are applied first alpha-numerically.Options inherited from parent commands--rootfs stringThe path to the 'real' host root filesystem. This will cause kubeadm to chroot into the provided path.SynopsisConfigures bootstrap token and cluster-info RBAC ruleskubeadm upgrade apply phase bootstrap-token [flags] Options--config stringPath to a kubeadm configuration file.--dry-runDo not change any state, just output what actions would be performed.-h, --helphelp for bootstrap-token--kubeconfig string Default: "/etc/kubernetes/admin.conf"The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations can be searched for an existing kubeconfig file.Options inherited from parent commands--rootfs stringThe path to the 'real' host root filesystem. This will cause kubeadm to chroot into the provided path.SynopsisUpgrade the default kubeadm addonskubeadm upgrade apply phase addon [flags] Options-h, --helphelp for addonOptions inherited from parent commands--rootfs stringThe path to the 'real' host root filesystem. This will cause kubeadm to chroot into the provided path.SynopsisRun post upgrade taskskubeadm upgrade apply phase post-upgrade [flags] Options--config stringPath to a kubeadm configuration file.--dry-runDo not change any state, just output what actions would be performed.-h, --helphelp for post-upgrade--kubeconfig string Default: "/etc/kubernetes/admin.conf"The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations can be searched for an existing kubeconfig file.Options inherited from parent commands--rootfs stringThe path to the 'real' host root filesystem. This will cause kubeadm to chroot into the provided path.kubeadm upgrade node phaseUsing the phases of kubeadm upgrade node you can choose to execute the separate steps of the upgrade of secondary control-plane or worker nodes.SynopsisUse this command to invoke single phase of the "node" workflowkubeadm upgrade node phase [flags] Options-h, --helphelp for phaseOptions inherited from parent commands--rootfs stringThe path to the 'real' host root filesystem. This will cause kubeadm to chroot into the provided path.Run upgrade node pre-flight checksSynopsisRun pre-flight checks for kubeadm upgrade node.kubeadm upgrade node phase preflight [flags] Options--config stringPath to a kubeadm configuration file.-h, --helphelp for preflight--ignore-preflight-errors stringsA list of checks whose errors will be shown as warnings. Example: 'IsPrivilegedUser,Swap'. Value 'all' ignores errors from all checks.Options inherited from parent commands--rootfs stringThe path to the 'real' host root filesystem. This will cause kubeadm to chroot into the provided path.SynopsisUpgrade the control plane instance deployed on this node, if anykubeadm upgrade node phase control-plane [flags] Options--certificate-renewal Default: truePerform the renewal of certificates used by component changed during upgrades.--config stringPath to a kubeadm configuration file.--dry-runDo not change any state, just output the actions that would be performed.--etcd-upgrade Default: truePerform the upgrade of etcd.-h, --helphelp for control-plane--kubeconfig string Default: "/etc/kubernetes/admin.conf"The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations can be searched for an existing kubeconfig file.--patches stringPath to a directory that contains files named "target[suffix][+patchtype].extension". For example, "kube-apiserver0+merge.yaml" or just "etcd.json". "target" can be one of "kube-apiserver", "kube-controller-manager", "kube-scheduler", "etcd", "kubeletconfiguration", "corednsdeployment". "patchtype" can be one of "strategic", "merge" or "json" and they match the patch formats supported by kubectl. The default "patchtype" is "strategic". "extension" must be either "json" or "yaml". "suffix" is an optional string that can be used to determine which patches are applied first alpha-numerically.Options inherited from parent commands--rootfs stringThe path to the 'real' host root filesystem. This will cause kubeadm to chroot into the provided path.SynopsisUpgrade the kubelet configuration for this node by downloading it from the kubelet-config ConfigMap stored in the clusterkubeadm upgrade node phase kubelet-config [flags] Options--config stringPath to a kubeadm configuration file.--dry-runDo not change any state, just output the actions that would be performed.-h, --helphelp for kubelet-config--kubeconfig string Default: "/etc/kubernetes/admin.conf"The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations can be searched for an existing kubeconfig file.--patches stringPath to a directory that contains files named "target[suffix][+patchtype].extension". For example, "kube-apiserver0+merge.yaml" or just "etcd.json". "target" can be one of "kube-apiserver", "kube-controller-manager", "kube-scheduler", "etcd", "kubeletconfiguration", "corednsdeployment". "patchtype" can be one of "strategic", "merge" or "json" and they match the patch formats supported by kubectl. The default "patchtype" is "strategic". "extension" must be either "json" or "yaml". "suffix" is an optional string that can be used to determine which patches are applied first alpha-numerically.Options inherited from parent commands--rootfs stringThe path to the 'real' host root filesystem. This will cause kubeadm to chroot into the provided path.SynopsisUpgrade the default kubeadm addonskubeadm upgrade node phase addon [flags] Options-h, --helphelp for addonOptions inherited from parent commands--rootfs stringThe path to the 'real' host root filesystem. This will cause kubeadm to chroot into the provided path.SynopsisRun post upgrade taskskubeadm upgrade node phase post-upgrade [flags] Options--config stringPath to a kubeadm configuration file.--dry-runDo not change any state, just output the actions that would be performed.-h, --helphelp for post-upgrade--kubeconfig string Default: "/etc/kubernetes/admin.conf"The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations can be searched for an existing kubeconfig file.Options inherited from parent commands--rootfs stringThe path to the 'real' host root filesystem. This will cause kubeadm to chroot into the provided path.What's nextkubeadm init to bootstrap a Kubernetes control-plane nodekubeadm join to connect a node to the clusterkubeadm reset to revert any changes made to this host by kubeadm init or kubeadm joinkubeadm upgrade to upgrade a kubeadm nodekubeadm alpha to try experimental functionality

kubeadm upgrade phases


Certificate Management with kubeadmFEATURE STATE: Kubernetes v1.15 [stable]Client certificates generated by kubeadm expire after 1 year. This page explains how to manage certificate renewals with kubeadm. It also covers other tasks related to kubeadm certificate management.The Kubernetes project recommends upgrading to the latest patch releases promptly, and to ensure that you are running a supported minor release of Kubernetes. Following this recommendation helps you to stay secure.Before you beginYou should be familiar with PKI certificates and requirements in Kubernetes.You should be familiar with how to pass a configuration file to the kubeadm commands.This guide covers the usage of the openssl command (used for manual certificate signing, if you choose that approach), but you can use your preferred tools.Some of the steps here use sudo for administrator access. You can use any equivalent tool.Using custom certificatesBy default, kubeadm generates all the certificates needed for a cluster to run. You can override this behavior by providing your own certificates.To do so, you must place them in whatever directory is specified by the --cert-dir flag or the certificatesDir field of kubeadm's ClusterConfiguration. By default this is /etc/kubernetes/pki.If a given certificate and private key pair exists before running kubeadm init, kubeadm does not overwrite them. This means you can, for example, copy an existing CA into /etc/kubernetes/pki/ca.crt and /etc/kubernetes/pki/ca.key, and kubeadm will use this CA for signing the rest of the certificates.Choosing an encryption algorithmkubeadm allows you to choose an encryption algorithm that is used for creating public and private keys. That can be done by using the encryptionAlgorithm field of the kubeadm configuration:apiVersion: kubeadm.k8s.io/v1beta4 kind: ClusterConfiguration encryptionAlgorithm: <ALGORITHM> <ALGORITHM> can be one of RSA-2048 (default), RSA-3072, RSA-4096 or ECDSA-P256.Choosing certificate validity periodkubeadm allows you to choose the validity period of CA and leaf certificates. That can be done by using the certificateValidityPeriod and caCertificateValidityPeriod fields of the kubeadm configuration:apiVersion: kubeadm.k8s.io/v1beta4 kind: ClusterConfiguration certificateValidityPeriod: 8760h # Default: 365 days × 24 hours = 1 year caCertificateValidityPeriod: 87600h # Default: 365 days × 24 hours * 10 = 10 years The values of the fields follow the accepted format for Go's time.Duration values, with the longest supported unit being h (hours).External CA modeIt is also possible to provide only the ca.crt file and not the ca.key file (this is only available for the root CA file, not other cert pairs). If all other certificates and kubeconfig files are in place, kubeadm recognizes this condition and activates the "External CA" mode. kubeadm will proceed without the CA key on disk.Instead, run the controller-manager standalone with --controllers=csrsigner and point to the CA certificate and key.There are various ways to prepare the component credentials when using external CA mode.Manual preparation of component credentialsPKI certificates and requirements includes information on how to prepare all the required by kubeadm component credentials manually.This guide covers the usage of the openssl command (used for manual certificate signing, if you choose that approach), but you can use your preferred tools.Preparation of credentials by signing CSRs generated by kubeadmkubeadm can generate CSR files that you can sign manually with tools like openssl and your external CA. These CSR files will include all the specification for credentials that components deployed by kubeadm require.Automated preparation of component credentials by using kubeadm phasesAlternatively, it is possible to use kubeadm phase commands to automate this process.Go to a host that you want to prepare as a kubeadm control plane node with external CA.Copy the external CA files ca.crt and ca.key that you have into /etc/kubernetes/pki on the node.Prepare a temporary kubeadm configuration file called config.yaml that can be used with kubeadm init. Make sure that this file includes any relevant cluster wide or host-specific information that could be included in certificates, such as, ClusterConfiguration.controlPlaneEndpoint, ClusterConfiguration.certSANs and InitConfiguration.APIEndpoint.On the same host execute the commands kubeadm init phase kubeconfig all --config config.yaml and kubeadm init phase certs all --config config.yaml. This will generate all required kubeconfig files and certificates under /etc/kubernetes/ and its pki sub directory.Inspect the generated files. Delete /etc/kubernetes/pki/ca.key, delete or move to a safe location the file /etc/kubernetes/super-admin.conf.On nodes where kubeadm join will be called also delete /etc/kubernetes/kubelet.conf. This file is only required on the first node where kubeadm init will be called.Note that some files such pki/sa.*, pki/front-proxy-ca.* and pki/etc/ca.* are shared between control plane nodes, You can generate them once and distribute them manually to nodes where kubeadm join will be called, or you can use the --upload-certs functionality of kubeadm init and --certificate-key of kubeadm join to automate this distribution.Once the credentials are prepared on all nodes, call kubeadm init and kubeadm join for these nodes to join the cluster. kubeadm will use the existing kubeconfig and certificate files under /etc/kubernetes/ and its pki sub directory.Certificate expiry and managementNote:kubeadm cannot manage certificates signed by an external CA.You can use the check-expiration subcommand to check when certificates expire:kubeadm certs check-expiration The output is similar to this:CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGED admin.conf Dec 30, 2020 23:36 UTC 364d no apiserver Dec 30, 2020 23:36 UTC 364d ca no apiserver-etcd-client Dec 30, 2020 23:36 UTC 364d etcd-ca no apiserver-kubelet-client Dec 30, 2020 23:36 UTC 364d ca no controller-manager.conf Dec 30, 2020 23:36 UTC 364d no etcd-healthcheck-client Dec 30, 2020 23:36 UTC 364d etcd-ca no etcd-peer Dec 30, 2020 23:36 UTC 364d etcd-ca no etcd-server Dec 30, 2020 23:36 UTC 364d etcd-ca no front-proxy-client Dec 30, 2020 23:36 UTC 364d front-proxy-ca no scheduler.conf Dec 30, 2020 23:36 UTC 364d no CERTIFICATE AUTHORITY EXPIRES RESIDUAL TIME EXTERNALLY MANAGED ca Dec 28, 2029 23:36 UTC 9y no etcd-ca Dec 28, 2029 23:36 UTC 9y no front-proxy-ca Dec 28, 2029 23:36 UTC 9y no The command shows expiration/residual time for the client certificates in the /etc/kubernetes/pki folder and for the client certificate embedded in the kubeconfig files used by kubeadm (admin.conf, controller-manager.conf and scheduler.conf).Additionally, kubeadm informs the user if the certificate is externally managed; in this case, the user should take care of managing certificate renewal manually/using other tools.The kubelet.conf configuration file is not included in the list above because kubeadm configures kubelet for automatic certificate renewal with rotatable certificates under /var/lib/kubelet/pki. To repair an expired kubelet client certificate see Kubelet client certificate rotation fails.Note:On nodes created with kubeadm init from versions prior to kubeadm version 1.17, there is a bug where you manually have to modify the contents of kubelet.conf. After kubeadm init finishes, you should update kubelet.conf to point to the rotated kubelet client certificates, by replacing client-certificate-data and client-key-data with:client-certificate: /var/lib/kubelet/pki/kubelet-client-current.pem client-key: /var/lib/kubelet/pki/kubelet-client-current.pem Automatic certificate renewalkubeadm renews all the certificates during control plane upgrade.This feature is designed for addressing the simplest use cases; if you don't have specific requirements on certificate renewal and perform Kubernetes version upgrades regularly (less than 1 year in between each upgrade), kubeadm will take care of keeping your cluster up to date and reasonably secure.If you have more complex requirements for certificate renewal, you can opt out from the default behavior by passing --certificate-renewal=false to kubeadm upgrade apply or to kubeadm upgrade node.Manual certificate renewalYou can renew your certificates manually at any time with the kubeadm certs renew command, with the appropriate command line options. If you are running cluster with a replicated control plane, this command needs to be executed on all the control-plane nodes.This command performs the renewal using CA (or front-proxy-CA) certificate and key stored in /etc/kubernetes/pki.kubeadm certs renew uses the existing certificates as the authoritative source for attributes (Common Name, Organization, subject alternative name) and does not rely on the kubeadm-config ConfigMap. Even so, the Kubernetes project recommends keeping the served certificate and the associated values in that ConfigMap synchronized, to avoid any risk of confusion.After running the command you should restart the control plane Pods. This is required since dynamic certificate reload is currently not supported for all components and certificates. Static Pods are managed by the local kubelet and not by the API Server, thus kubectl cannot be used to delete and restart them. To restart a static Pod you can temporarily remove its manifest file from /etc/kubernetes/manifests/ and wait for 20 seconds (see the fileCheckFrequency value in KubeletConfiguration struct). The kubelet will terminate the Pod if it's no longer in the manifest directory. You can then move the file back and after another fileCheckFrequency period, the kubelet will recreate the Pod and the certificate renewal for the component can complete.kubeadm certs renew can renew any specific certificate or, with the subcommand all, it can renew all of them:# If you are running cluster with a replicated control plane, this command # needs to be executed on all the control-plane nodes. kubeadm certs renew all Copying the administrator certificate (optional)Clusters built with kubeadm often copy the admin.conf certificate into $HOME/.kube/config, as instructed in Creating a cluster with kubeadm. On such a system, to update the contents of $HOME/.kube/config after renewing the admin.conf, you could run the following commands:sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config Renew certificates with the Kubernetes certificates APIThis section provides more details about how to execute manual certificate renewal using the Kubernetes certificates API.Caution:These are advanced topics for users who need to integrate their organization's certificate infrastructure into a kubeadm-built cluster. If the default kubeadm configuration satisfies your needs, you should let kubeadm manage certificates instead.Set up a signerThe Kubernetes Certificate Authority does not work out of the box. You can configure an external signer such as cert-manager, or you can use the built-in signer.The built-in signer is part of kube-controller-manager.To activate the built-in signer, you must pass the --cluster-signing-cert-file and --cluster-signing-key-file flags.If you're creating a new cluster, you can use a kubeadm configuration file:apiVersion: kubeadm.k8s.io/v1beta4 kind: ClusterConfiguration controllerManager: extraArgs: - name: "cluster-signing-cert-file" value: "/etc/kubernetes/pki/ca.crt" - name: "cluster-signing-key-file" value: "/etc/kubernetes/pki/ca.key" Create certificate signing requests (CSR)See Create CertificateSigningRequest for creating CSRs with the Kubernetes API.Renew certificates with external CAThis section provide more details about how to execute manual certificate renewal using an external CA.To better integrate with external CAs, kubeadm can also produce certificate signing requests (CSRs). A CSR represents a request to a CA for a signed certificate for a client. In kubeadm terms, any certificate that would normally be signed by an on-disk CA can be produced as a CSR instead. A CA, however, cannot be produced as a CSR.Renewal by using certificate signing requests (CSR)Renewal of ceritficates is possible by generating new CSRs and signing them with the external CA. For more details about working with CSRs generated by kubeadm see the section Signing certificate signing requests (CSR) generated by kubeadm.Certificate authority (CA) rotationKubeadm does not support rotation or replacement of CA certificates out of the box.For more information about manual rotation or replacement of CA, see manual rotation of CA certificates.Enabling signed kubelet serving certificatesBy default the kubelet serving certificate deployed by kubeadm is self-signed. This means a connection from external services like the metrics-server to a kubelet cannot be secured with TLS.To configure the kubelets in a new kubeadm cluster to obtain properly signed serving certificates you must pass the following minimal configuration to kubeadm init:apiVersion: kubeadm.k8s.io/v1beta4 kind: ClusterConfiguration --- apiVersion: kubelet.config.k8s.io/v1beta1 kind: KubeletConfiguration serverTLSBootstrap: true If you have already created the cluster you must adapt it by doing the following:Find and edit the kubelet-config ConfigMap in the kube-system namespace. In that ConfigMap, the kubelet key has a KubeletConfiguration document as its value. Edit the KubeletConfiguration document to set serverTLSBootstrap: true.On each node, add the serverTLSBootstrap: true field in /var/lib/kubelet/config.yaml and restart the kubelet with systemctl restart kubeletThe field serverTLSBootstrap: true will enable the bootstrap of kubelet serving certificates by requesting them from the certificates.k8s.io API. One known limitation is that the CSRs (Certificate Signing Requests) for these certificates cannot be automatically approved by the default signer in the kube-controller-manager - kubernetes.io/kubelet-serving. This will require action from the user or a third party controller.These CSRs can be viewed using:kubectl get csr NAME AGE SIGNERNAME REQUESTOR CONDITION csr-9wvgt 112s kubernetes.io/kubelet-serving system:node:worker-1 Pending csr-lz97v 1m58s kubernetes.io/kubelet-serving system:node:control-plane-1 Pending To approve them you can do the following:kubectl certificate approve <CSR-name> By default, these serving certificate will expire after one year. Kubeadm sets the KubeletConfiguration field rotateCertificates to true, which means that close to expiration a new set of CSRs for the serving certificates will be created and must be approved to complete the rotation. To understand more see Certificate Rotation.If you are looking for a solution for automatic approval of these CSRs it is recommended that you contact your cloud provider and ask if they have a CSR signer that verifies the node identity with an out of band mechanism.Note: This section links to third party projects that provide functionality required by Kubernetes. The Kubernetes project authors aren't responsible for these projects, which are listed alphabetically. To add a project to this list, read the content guide before submitting a change. More information.Third party custom controllers can be used:kubelet-csr-approverSuch a controller is not a secure mechanism unless it not only verifies the CommonName in the CSR but also verifies the requested IPs and domain names. This would prevent a malicious actor that has access to a kubelet client certificate to create CSRs requesting serving certificates for any IP or domain name.Generating kubeconfig files for additional usersDuring cluster creation, kubeadm init signs the certificate in the super-admin.conf to have Subject: O = system:masters, CN = kubernetes-super-admin. system:masters is a break-glass, super user group that bypasses the authorization layer (for example, RBAC). The file admin.conf is also created by kubeadm on control plane nodes and it contains a certificate with Subject: O = kubeadm:cluster-admins, CN = kubernetes-admin. kubeadm:cluster-admins is a group logically belonging to kubeadm. If your cluster uses RBAC (the kubeadm default), the kubeadm:cluster-admins group is bound to the cluster-admin ClusterRole.Warning:Avoid sharing the super-admin.conf or admin.conf files. Instead, create least privileged access even for people who work as administrators and use that least privilege alternative for anything other than break-glass (emergency) access.You can use the kubeadm kubeconfig user command to generate kubeconfig files for additional users. The command accepts a mixture of command line flags and kubeadm configuration options. The generated kubeconfig will be written to stdout and can be piped to a file using kubeadm kubeconfig user ... > somefile.conf.Example configuration file that can be used with --config:# example.yaml apiVersion: kubeadm.k8s.io/v1beta4 kind: ClusterConfiguration # Will be used as the target "cluster" in the kubeconfig clusterName: "kubernetes" # Will be used as the "server" (IP or DNS name) of this cluster in the kubeconfig controlPlaneEndpoint: "some-dns-address:6443" # The cluster CA key and certificate will be loaded from this local directory certificatesDir: "/etc/kubernetes/pki" Make sure that these settings match the desired target cluster settings. To see the settings of an existing cluster use:kubectl get cm kubeadm-config -n kube-system -o=jsonpath="{.data.ClusterConfiguration}" The following example will generate a kubeconfig file with credentials valid for 24 hours for a new user johndoe that is part of the appdevs group:kubeadm kubeconfig user --config example.yaml --org appdevs --client-name johndoe --validity-period 24h The following example will generate a kubeconfig file with administrator credentials valid for 1 week:kubeadm kubeconfig user --config example.yaml --client-name admin --validity-period 168h Signing certificate signing requests (CSR) generated by kubeadmYou can create certificate signing requests with kubeadm certs generate-csr. Calling this command will generate .csr / .key file pairs for regular certificates. For certificates embedded in kubeconfig files, the command will generate a .csr / .conf pair where the key is already embedded in the .conf file.A CSR file contains all relevant information for a CA to sign a certificate. kubeadm uses a well defined specification for all its certificates and CSRs.The default certificate directory is /etc/kubernetes/pki, while the default directory for kubeconfig files is /etc/kubernetes. These defaults can be overridden with the flags --cert-dir and --kubeconfig-dir, respectively.To pass custom options to kubeadm certs generate-csr use the --config flag, which accepts a kubeadm configuration file, similarly to commands such as kubeadm init. Any specification such as extra SANs and custom IP addresses must be stored in the same configuration file and used for all relevant kubeadm commands by passing it as --config.Note:This guide uses the default Kubernetes directory /etc/kubernetes, which requires a super user. If you are following this guide and are using directories that you can write to (typically, this means running kubeadm with --cert-dir and --kubeconfig-dir) then you can omit the sudo command.You must then copy the files that you produced over to within the /etc/kubernetes directory so that kubeadm init or kubeadm join will find them.Preparing CA and service account filesOn the primary control plane node, where kubeadm init will be executed, call the following commands:sudo kubeadm init phase certs ca sudo kubeadm init phase certs etcd-ca sudo kubeadm init phase certs front-proxy-ca sudo kubeadm init phase certs sa This will populate the folders /etc/kubernetes/pki and /etc/kubernetes/pki/etcd with all self-signed CA files (certificates and keys) and service account (public and private keys) that kubeadm needs for a control plane node.Note:If you are using an external CA, you must generate the same files out of band and manually copy them to the primary control plane node in /etc/kubernetes.Once all CSRs are signed, you can delete the root CA key (ca.key) as noted in the External CA mode section.For secondary control plane nodes (kubeadm join --control-plane) there is no need to call the above commands. Depending on how you setup the High Availability cluster, you either have to manually copy the same files from the primary control plane node, or use the automated --upload-certs functionality of kubeadm init.Generate CSRsThe kubeadm certs generate-csr command generates CSRs for all known certificates managed by kubeadm. Once the command is done you must manually delete .csr, .conf or .key files that you don't need.Considerations for kubelet.confThis section applies to both control plane and worker nodes.If you have deleted the ca.key file from control plane nodes (External CA mode), the active kube-controller-manager in this cluster will not be able to sign kubelet client certificates. If no external method for signing these certificates exists in your setup (such as an external signer), you could manually sign the kubelet.conf.csr as explained in this guide.Note that this also means that the automatic kubelet client certificate rotation will be disabled. If so, close to certificate expiration, you must generate a new kubelet.conf.csr, sign the certificate, embed it in kubelet.conf and restart the kubelet.If this does not apply to your setup, you can skip processing the kubelet.conf.csr on secondary control plane and on workers nodes (all nodes that call kubeadm join ...). That is because the active kube-controller-manager will be responsible for signing new kubelet client certificates.Note:You must process the kubelet.conf.csr file on the primary control plane node (the host where you originally ran kubeadm init). This is because kubeadm considers that as the node that bootstraps the cluster, and a pre-populated kubelet.conf is needed.Control plane nodesExecute the following command on primary (kubeadm init) and secondary (kubeadm join --control-plane) control plane nodes to generate all CSR files:sudo kubeadm certs generate-csr If external etcd is to be used, follow the External etcd with kubeadm guide to understand what CSR files are needed on the kubeadm and etcd nodes. Other .csr and .key files under /etc/kubernetes/pki/etcd can be removed.Based on the explanation in Considerations for kubelet.conf keep or delete the kubelet.conf and kubelet.conf.csr files.Worker nodesBased on the explanation in Considerations for kubelet.conf, optionally call:sudo kubeadm certs generate-csr and keep only the kubelet.conf and kubelet.conf.csr files. Alternatively skip the steps for worker nodes entirely.Signing CSRs for all certificatesNote:If you are using external CA and already have CA serial number files (.srl) for openssl, you can copy such files to a kubeadm node where CSRs will be processed. The .srl files to copy are /etc/kubernetes/pki/ca.srl, /etc/kubernetes/pki/front-proxy-ca.srl and /etc/kubernetes/pki/etcd/ca.srl. The files can be then moved to a new node where CSR files will be processed.If a .srl file is missing for a CA on a node, the script below will generate a new SRL file with a random starting serial number.To read more about .srl files see the openssl documentation for the --CAserial flag.Repeat this step for all nodes that have CSR files.Write the following script in the /etc/kubernetes directory, navigate to the directory and execute the script. The script will generate certificates for all CSR files that are present in the /etc/kubernetes tree.#!/bin/bash # Set certificate expiration time in days DAYS=365 # Process all CSR files except those for front-proxy and etcd find ./ -name "*.csr" | grep -v "pki/etcd" | grep -v "front-proxy" | while read -r FILE; do echo "* Processing ${FILE} ..." FILE=${FILE%.*} # Trim the extension if [ -f "./pki/ca.srl" ]; then SERIAL_FLAG="-CAserial ./pki/ca.srl" else SERIAL_FLAG="-CAcreateserial" fi openssl x509 -req -days "${DAYS}" -CA ./pki/ca.crt -CAkey ./pki/ca.key ${SERIAL_FLAG} \ -in "${FILE}.csr" -out "${FILE}.crt" sleep 2 done # Process all etcd CSRs find ./pki/etcd -name "*.csr" | while read -r FILE; do echo "* Processing ${FILE} ..." FILE=${FILE%.*} # Trim the extension if [ -f "./pki/etcd/ca.srl" ]; then SERIAL_FLAG=-CAserial ./pki/etcd/ca.srl else SERIAL_FLAG=-CAcreateserial fi openssl x509 -req -days "${DAYS}" -CA ./pki/etcd/ca.crt -CAkey ./pki/etcd/ca.key ${SERIAL_FLAG} \ -in "${FILE}.csr" -out "${FILE}.crt" done # Process front-proxy CSRs echo "* Processing ./pki/front-proxy-client.csr ..." openssl x509 -req -days "${DAYS}" -CA ./pki/front-proxy-ca.crt -CAkey ./pki/front-proxy-ca.key -CAcreateserial \ -in ./pki/front-proxy-client.csr -out ./pki/front-proxy-client.crt Embedding certificates in kubeconfig filesRepeat this step for all nodes that have CSR files.Write the following script in the /etc/kubernetes directory, navigate to the directory and execute the script. The script will take the .crt files that were signed for kubeconfig files from CSRs in the previous step and will embed them in the kubeconfig files.#!/bin/bash CLUSTER=kubernetes find ./ -name "*.conf" | while read -r FILE; do echo "* Processing ${FILE} ..." KUBECONFIG="${FILE}" kubectl config set-cluster "${CLUSTER}" --certificate-authority ./pki/ca.crt --embed-certs USER=$(KUBECONFIG="${FILE}" kubectl config view -o jsonpath='{.users[0].name}') KUBECONFIG="${FILE}" kubectl config set-credentials "${USER}" --client-certificate "${FILE}.crt" --embed-certs done Performing cleanupPerform this step on all nodes that have CSR files.Write the following script in the /etc/kubernetes directory, navigate to the directory and execute the script.#!/bin/bash # Cleanup CSR files rm -f ./*.csr ./pki/*.csr ./pki/etcd/*.csr # Clean all CSR files # Cleanup CRT files that were already embedded in kubeconfig files rm -f ./*.crt Optionally, move .srl files to the next node to be processed.Optionally, if using external CA remove the /etc/kubernetes/pki/ca.key file, as explained in the External CA node section.kubeadm node initializationOnce CSR files have been signed and required certificates are in place on the hosts you want to use as nodes, you can use the commands kubeadm init and kubeadm join to create a Kubernetes cluster from these nodes. During init and join, kubeadm uses existing certificates, encryption keys and kubeconfig files that it finds in the /etc/kubernetes tree on the host's local filesystem.Items on this page refer to third party products or projects that provide functionality required by Kubernetes. The Kubernetes project authors aren't responsible for those third-party products or projects. See the CNCF website guidelines for more details.You should read the content guide before proposing a change that adds an extra third-party link.

Certificate Management with kubeadm


Set up a High Availability etcd Cluster with kubeadmBy default, kubeadm runs a local etcd instance on each control plane node. It is also possible to treat the etcd cluster as external and provision etcd instances on separate hosts. The differences between the two approaches are covered in the Options for Highly Available topology page.This task walks through the process of creating a high availability external etcd cluster of three members that can be used by kubeadm during cluster creation.Before you beginThree hosts that can talk to each other over TCP ports 2379 and 2380. This document assumes these default ports. However, they are configurable through the kubeadm config file.Each host must have systemd and a bash compatible shell installed.Each host must have a container runtime, kubelet, and kubeadm installed.Each host should have access to the Kubernetes container image registry (registry.k8s.io) or list/pull the required etcd image using kubeadm config images list/pull. This guide will set up etcd instances as static pods managed by a kubelet.Some infrastructure to copy files between hosts. For example ssh and scp can satisfy this requirement.Setting up the clusterThe general approach is to generate all certs on one node and only distribute the necessary files to the other nodes.Note:kubeadm contains all the necessary cryptographic machinery to generate the certificates described below; no other cryptographic tooling is required for this example.Note:The examples below use IPv4 addresses but you can also configure kubeadm, the kubelet and etcd to use IPv6 addresses. Dual-stack is supported by some Kubernetes options, but not by etcd. For more details on Kubernetes dual-stack support see Dual-stack support with kubeadm.Configure the kubelet to be a service manager for etcd.Note:You must do this on every host where etcd should be running.Since etcd was created first, you must override the service priority by creating a new unit file that has higher precedence than the kubeadm-provided kubelet unit file.cat << EOF > /etc/systemd/system/kubelet.service.d/kubelet.conf # Replace "systemd" with the cgroup driver of your container runtime. The default value in the kubelet is "cgroupfs". # Replace the value of "containerRuntimeEndpoint" for a different container runtime if needed. # apiVersion: kubelet.config.k8s.io/v1beta1 kind: KubeletConfiguration authentication: anonymous: enabled: false webhook: enabled: false authorization: mode: AlwaysAllow cgroupDriver: systemd address: 127.0.0.1 containerRuntimeEndpoint: unix:///var/run/containerd/containerd.sock staticPodPath: /etc/kubernetes/manifests EOF cat << EOF > /etc/systemd/system/kubelet.service.d/20-etcd-service-manager.conf [Service] ExecStart= ExecStart=/usr/bin/kubelet --config=/etc/systemd/system/kubelet.service.d/kubelet.conf Restart=always EOF systemctl daemon-reload systemctl restart kubelet Check the kubelet status to ensure it is running.systemctl status kubelet Create configuration files for kubeadm.Generate one kubeadm configuration file for each host that will have an etcd member running on it using the following script.# Update HOST0, HOST1 and HOST2 with the IPs of your hosts export HOST0=10.0.0.6 export HOST1=10.0.0.7 export HOST2=10.0.0.8 # Update NAME0, NAME1 and NAME2 with the hostnames of your hosts export NAME0="infra0" export NAME1="infra1" export NAME2="infra2" # Create temp directories to store files that will end up on other hosts mkdir -p /tmp/${HOST0}/ /tmp/${HOST1}/ /tmp/${HOST2}/ HOSTS=(${HOST0} ${HOST1} ${HOST2}) NAMES=(${NAME0} ${NAME1} ${NAME2}) for i in "${!HOSTS[@]}"; do HOST=${HOSTS[$i]} NAME=${NAMES[$i]} cat << EOF > /tmp/${HOST}/kubeadmcfg.yaml --- apiVersion: "kubeadm.k8s.io/v1beta4" kind: InitConfiguration nodeRegistration: name: ${NAME} localAPIEndpoint: advertiseAddress: ${HOST} --- apiVersion: "kubeadm.k8s.io/v1beta4" kind: ClusterConfiguration etcd: local: serverCertSANs: - "${HOST}" peerCertSANs: - "${HOST}" extraArgs: - name: initial-cluster value: ${NAMES[0]}=https://${HOSTS[0]}:2380,${NAMES[1]}=https://${HOSTS[1]}:2380,${NAMES[2]}=https://${HOSTS[2]}:2380 - name: initial-cluster-state value: new - name: name value: ${NAME} - name: listen-peer-urls value: https://${HOST}:2380 - name: listen-client-urls value: https://${HOST}:2379 - name: advertise-client-urls value: https://${HOST}:2379 - name: initial-advertise-peer-urls value: https://${HOST}:2380 EOF done Generate the certificate authority.If you already have a CA then the only action that is copying the CA's crt and key file to /etc/kubernetes/pki/etcd/ca.crt and /etc/kubernetes/pki/etcd/ca.key. After those files have been copied, proceed to the next step, "Create certificates for each member".If you do not already have a CA then run this command on $HOST0 (where you generated the configuration files for kubeadm).kubeadm init phase certs etcd-ca This creates two files:/etc/kubernetes/pki/etcd/ca.crt/etc/kubernetes/pki/etcd/ca.keyCreate certificates for each member.kubeadm init phase certs etcd-server --config=/tmp/${HOST2}/kubeadmcfg.yaml kubeadm init phase certs etcd-peer --config=/tmp/${HOST2}/kubeadmcfg.yaml kubeadm init phase certs etcd-healthcheck-client --config=/tmp/${HOST2}/kubeadmcfg.yaml kubeadm init phase certs apiserver-etcd-client --config=/tmp/${HOST2}/kubeadmcfg.yaml cp -R /etc/kubernetes/pki /tmp/${HOST2}/ # cleanup non-reusable certificates find /etc/kubernetes/pki -not -name ca.crt -not -name ca.key -type f -delete kubeadm init phase certs etcd-server --config=/tmp/${HOST1}/kubeadmcfg.yaml kubeadm init phase certs etcd-peer --config=/tmp/${HOST1}/kubeadmcfg.yaml kubeadm init phase certs etcd-healthcheck-client --config=/tmp/${HOST1}/kubeadmcfg.yaml kubeadm init phase certs apiserver-etcd-client --config=/tmp/${HOST1}/kubeadmcfg.yaml cp -R /etc/kubernetes/pki /tmp/${HOST1}/ find /etc/kubernetes/pki -not -name ca.crt -not -name ca.key -type f -delete kubeadm init phase certs etcd-server --config=/tmp/${HOST0}/kubeadmcfg.yaml kubeadm init phase certs etcd-peer --config=/tmp/${HOST0}/kubeadmcfg.yaml kubeadm init phase certs etcd-healthcheck-client --config=/tmp/${HOST0}/kubeadmcfg.yaml kubeadm init phase certs apiserver-etcd-client --config=/tmp/${HOST0}/kubeadmcfg.yaml # No need to move the certs because they are for HOST0 # clean up certs that should not be copied off this host find /tmp/${HOST2} -name ca.key -type f -delete find /tmp/${HOST1} -name ca.key -type f -delete Copy certificates and kubeadm configs.The certificates have been generated and now they must be moved to their respective hosts.USER=ubuntu HOST=${HOST1} scp -r /tmp/${HOST}/* ${USER}@${HOST}: ssh ${USER}@${HOST} USER@HOST $ sudo -Es root@HOST $ chown -R root:root pki root@HOST $ mv pki /etc/kubernetes/ Ensure all expected files exist.The complete list of required files on $HOST0 is:/tmp/${HOST0} └── kubeadmcfg.yaml --- /etc/kubernetes/pki ├── apiserver-etcd-client.crt ├── apiserver-etcd-client.key └── etcd ├── ca.crt ├── ca.key ├── healthcheck-client.crt ├── healthcheck-client.key ├── peer.crt ├── peer.key ├── server.crt └── server.key On $HOST1:$HOME └── kubeadmcfg.yaml --- /etc/kubernetes/pki ├── apiserver-etcd-client.crt ├── apiserver-etcd-client.key └── etcd ├── ca.crt ├── healthcheck-client.crt ├── healthcheck-client.key ├── peer.crt ├── peer.key ├── server.crt └── server.key On $HOST2:$HOME └── kubeadmcfg.yaml --- /etc/kubernetes/pki ├── apiserver-etcd-client.crt ├── apiserver-etcd-client.key └── etcd ├── ca.crt ├── healthcheck-client.crt ├── healthcheck-client.key ├── peer.crt ├── peer.key ├── server.crt └── server.key Create the static pod manifests.Now that the certificates and configs are in place it's time to create the manifests. On each host run the kubeadm command to generate a static manifest for etcd.root@HOST0 $ kubeadm init phase etcd local --config=/tmp/${HOST0}/kubeadmcfg.yaml root@HOST1 $ kubeadm init phase etcd local --config=$HOME/kubeadmcfg.yaml root@HOST2 $ kubeadm init phase etcd local --config=$HOME/kubeadmcfg.yaml Optional: Check the cluster health.If etcdctl isn't available, you can run this tool inside a container image. You would do that directly with your container runtime using a tool such as crictl run and not through KubernetesETCDCTL_API=3 etcdctl \ --cert /etc/kubernetes/pki/etcd/peer.crt \ --key /etc/kubernetes/pki/etcd/peer.key \ --cacert /etc/kubernetes/pki/etcd/ca.crt \ --endpoints https://${HOST0}:2379 endpoint health ... https://[HOST0 IP]:2379 is healthy: successfully committed proposal: took = 16.283339ms https://[HOST1 IP]:2379 is healthy: successfully committed proposal: took = 19.44402ms https://[HOST2 IP]:2379 is healthy: successfully committed proposal: took = 35.926451ms Set ${HOST0}to the IP address of the host you are testing.What's nextOnce you have an etcd cluster with 3 working members, you can continue setting up a highly available control plane using the external etcd method with kubeadm.


Configuration Best PracticesThis document highlights and consolidates configuration best practices that are introduced throughout the user guide, Getting Started documentation, and examples.This is a living document. If you think of something that is not on this list but might be useful to others, please don't hesitate to file an issue or submit a PR.General Configuration TipsWhen defining configurations, specify the latest stable API version.Configuration files should be stored in version control before being pushed to the cluster. This allows you to quickly roll back a configuration change if necessary. It also aids cluster re-creation and restoration.Write your configuration files using YAML rather than JSON. Though these formats can be used interchangeably in almost all scenarios, YAML tends to be more user-friendly.Group related objects into a single file whenever it makes sense. One file is often easier to manage than several. See the guestbook-all-in-one.yaml file as an example of this syntax.Note also that many kubectl commands can be called on a directory. For example, you can call kubectl apply on a directory of config files.Don't specify default values unnecessarily: simple, minimal configuration will make errors less likely.Put object descriptions in annotations, to allow better introspection.Note:There is a breaking change introduced in the YAML 1.2 boolean values specification with respect to YAML 1.1. This is a known issue in Kubernetes. YAML 1.2 only recognizes true and false as valid booleans, while YAML 1.1 also accepts yes, no, on, and off as booleans. However, Kubernetes uses YAML parsers that are mostly compatible with YAML 1.1, which means that using yes or no instead of true or false in a YAML manifest may cause unexpected errors or behaviors. To avoid this issue, it is recommended to always use true or false for boolean values in YAML manifests, and to quote any strings that may be confused with booleans, such as "yes" or "no".Besides booleans, there are additional specifications changes between YAML versions. Please refer to the YAML Specification Changes documentation for a comprehensive list."Naked" Pods versus ReplicaSets, Deployments, and JobsDon't use naked Pods (that is, Pods not bound to a ReplicaSet or Deployment) if you can avoid it. Naked Pods will not be rescheduled in the event of a node failure.A Deployment, which both creates a ReplicaSet to ensure that the desired number of Pods is always available, and specifies a strategy to replace Pods (such as RollingUpdate), is almost always preferable to creating Pods directly, except for some explicit restartPolicy: Never scenarios. A Job may also be appropriate.ServicesCreate a Service before its corresponding backend workloads (Deployments or ReplicaSets), and before any workloads that need to access it. When Kubernetes starts a container, it provides environment variables pointing to all the Services which were running when the container was started. For example, if a Service named foo exists, all containers will get the following variables in their initial environment:FOO_SERVICE_HOST=<the host the Service is running on> FOO_SERVICE_PORT=<the port the Service is running on> This does imply an ordering requirement - any Service that a Pod wants to access must be created before the Pod itself, or else the environment variables will not be populated. DNS does not have this restriction.An optional (though strongly recommended) cluster add-on is a DNS server. The DNS server watches the Kubernetes API for new Services and creates a set of DNS records for each. If DNS has been enabled throughout the cluster then all Pods should be able to do name resolution of Services automatically.Don't specify a hostPort for a Pod unless it is absolutely necessary. When you bind a Pod to a hostPort, it limits the number of places the Pod can be scheduled, because each <hostIP, hostPort, protocol> combination must be unique. If you don't specify the hostIP and protocol explicitly, Kubernetes will use 0.0.0.0 as the default hostIP and TCP as the default protocol.If you only need access to the port for debugging purposes, you can use the apiserver proxy or kubectl port-forward.If you explicitly need to expose a Pod's port on the node, consider using a NodePort Service before resorting to hostPort.Avoid using hostNetwork, for the same reasons as hostPort.Use headless Services (which have a ClusterIP of None) for service discovery when you don't need kube-proxy load balancing.Using LabelsDefine and use labels that identify semantic attributes of your application or Deployment, such as { app.kubernetes.io/name: MyApp, tier: frontend, phase: test, deployment: v3 }. You can use these labels to select the appropriate Pods for other resources; for example, a Service that selects all tier: frontend Pods, or all phase: test components of app.kubernetes.io/name: MyApp. See the guestbook app for examples of this approach.A Service can be made to span multiple Deployments by omitting release-specific labels from its selector. When you need to update a running service without downtime, use a Deployment.A desired state of an object is described by a Deployment, and if changes to that spec are applied, the deployment controller changes the actual state to the desired state at a controlled rate.Use the Kubernetes common labels for common use cases. These standardized labels enrich the metadata in a way that allows tools, including kubectl and dashboard, to work in an interoperable way.You can manipulate labels for debugging. Because Kubernetes controllers (such as ReplicaSet) and Services match to Pods using selector labels, removing the relevant labels from a Pod will stop it from being considered by a controller or from being served traffic by a Service. If you remove the labels of an existing Pod, its controller will create a new Pod to take its place. This is a useful way to debug a previously "live" Pod in a "quarantine" environment. To interactively remove or add labels, use kubectl label.Using kubectlUse kubectl apply -f <directory>. This looks for Kubernetes configuration in all .yaml, .yml, and .json files in <directory> and passes it to apply.Use label selectors for get and delete operations instead of specific object names. See the sections on label selectors and using labels effectively.Use kubectl create deployment and kubectl expose to quickly create single-container Deployments and Services. See Use a Service to Access an Application in a Cluster for an example.

Configuration Best Practices