SubjectAccessReviewSubjectAccessReview checks whether or not a user or group can perform an action.apiVersion: authorization.k8s.io/v1import "k8s.io/api/authorization/v1"SubjectAccessReviewSubjectAccessReview checks whether or not a user or group can perform an action.apiVersion: authorization.k8s.io/v1kind: SubjectAccessReviewmetadata (ObjectMeta)Standard list metadata. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadataspec (SubjectAccessReviewSpec), requiredSpec holds information about the request being evaluatedstatus (SubjectAccessReviewStatus)Status is filled in by the server and indicates whether the request is allowed or notSubjectAccessReviewSpecSubjectAccessReviewSpec is a description of the access request. Exactly one of ResourceAuthorizationAttributes and NonResourceAuthorizationAttributes must be setextra (map[string][]string)Extra corresponds to the user.Info.GetExtra() method from the authenticator. Since that is input to the authorizer it needs a reflection here.groups ([]string)Atomic: will be replaced during a mergeGroups is the groups you're testing for.nonResourceAttributes (NonResourceAttributes)NonResourceAttributes describes information for a non-resource access requestNonResourceAttributes includes the authorization attributes available for non-resource requests to the Authorizer interfacenonResourceAttributes.path (string)Path is the URL path of the requestnonResourceAttributes.verb (string)Verb is the standard HTTP verbresourceAttributes (ResourceAttributes)ResourceAuthorizationAttributes describes information for a resource access requestResourceAttributes includes the authorization attributes available for resource requests to the Authorizer interfaceresourceAttributes.fieldSelector (FieldSelectorAttributes)fieldSelector describes the limitation on access based on field. It can only limit access, not broaden it.*FieldSelectorAttributes indicates a field limited access. Webhook authors are encouraged to * ensure rawSelector and requirements are not both set * consider the requirements field if set * not try to parse or consider the rawSelector field if set. This is to avoid another CVE-2022-2880 (i.e. getting different systems to agree on how exactly to parse a query is not something we want), see https://www.oxeye.io/resources/golang-parameter-smuggling-attack for more details. For the SubjectAccessReview endpoints of the kube-apiserver: * If rawSelector is empty and requirements are empty, the request is not limited. * If rawSelector is present and requirements are empty, the rawSelector will be parsed and limited if the parsing succeeds. * If rawSelector is empty and requirements are present, the requirements should be honored * If rawSelector is present and requirements are present, the request is invalid.resourceAttributes.fieldSelector.rawSelector (string)rawSelector is the serialization of a field selector that would be included in a query parameter. Webhook implementations are encouraged to ignore rawSelector. The kube-apiserver's *SubjectAccessReview will parse the rawSelector as long as the requirements are not present.resourceAttributes.fieldSelector.requirements ([]FieldSelectorRequirement)Atomic: will be replaced during a mergerequirements is the parsed interpretation of a field selector. All requirements must be met for a resource instance to match the selector. Webhook implementations should handle requirements, but how to handle them is up to the webhook. Since requirements can only limit the request, it is safe to authorize as unlimited request if the requirements are not understood.FieldSelectorRequirement is a selector that contains values, a key, and an operator that relates the key and values.resourceAttributes.fieldSelector.requirements.key (string), requiredkey is the field selector key that the requirement applies to.resourceAttributes.fieldSelector.requirements.operator (string), requiredoperator represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists, DoesNotExist. The list of operators may grow in the future.resourceAttributes.fieldSelector.requirements.values ([]string)Atomic: will be replaced during a mergevalues is an array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty.resourceAttributes.group (string)Group is the API Group of the Resource. "*" means all.resourceAttributes.labelSelector (LabelSelectorAttributes)labelSelector describes the limitation on access based on labels. It can only limit access, not broaden it.*LabelSelectorAttributes indicates a label limited access. Webhook authors are encouraged to * ensure rawSelector and requirements are not both set * consider the requirements field if set * not try to parse or consider the rawSelector field if set. This is to avoid another CVE-2022-2880 (i.e. getting different systems to agree on how exactly to parse a query is not something we want), see https://www.oxeye.io/resources/golang-parameter-smuggling-attack for more details. For the SubjectAccessReview endpoints of the kube-apiserver: * If rawSelector is empty and requirements are empty, the request is not limited. * If rawSelector is present and requirements are empty, the rawSelector will be parsed and limited if the parsing succeeds. * If rawSelector is empty and requirements are present, the requirements should be honored * If rawSelector is present and requirements are present, the request is invalid.resourceAttributes.labelSelector.rawSelector (string)rawSelector is the serialization of a field selector that would be included in a query parameter. Webhook implementations are encouraged to ignore rawSelector. The kube-apiserver's *SubjectAccessReview will parse the rawSelector as long as the requirements are not present.resourceAttributes.labelSelector.requirements ([]LabelSelectorRequirement)Atomic: will be replaced during a mergerequirements is the parsed interpretation of a label selector. All requirements must be met for a resource instance to match the selector. Webhook implementations should handle requirements, but how to handle them is up to the webhook. Since requirements can only limit the request, it is safe to authorize as unlimited request if the requirements are not understood.A label selector requirement is a selector that contains values, a key, and an operator that relates the key and values.resourceAttributes.labelSelector.requirements.key (string), requiredkey is the label key that the selector applies to.resourceAttributes.labelSelector.requirements.operator (string), requiredoperator represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists and DoesNotExist.resourceAttributes.labelSelector.requirements.values ([]string)Atomic: will be replaced during a mergevalues is an array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. This array is replaced during a strategic merge patch.resourceAttributes.name (string)Name is the name of the resource being requested for a "get" or deleted for a "delete". "" (empty) means all.resourceAttributes.namespace (string)Namespace is the namespace of the action being requested. Currently, there is no distinction between no namespace and all namespaces "" (empty) is defaulted for LocalSubjectAccessReviews "" (empty) is empty for cluster-scoped resources "" (empty) means "all" for namespace scoped resources from a SubjectAccessReview or SelfSubjectAccessReviewresourceAttributes.resource (string)Resource is one of the existing resource types. "*" means all.resourceAttributes.subresource (string)Subresource is one of the existing resource types. "" means none.resourceAttributes.verb (string)Verb is a kubernetes resource API verb, like: get, list, watch, create, update, delete, proxy. "*" means all.resourceAttributes.version (string)Version is the API Version of the Resource. "*" means all.uid (string)UID information about the requesting user.user (string)User is the user you're testing for. If you specify "User" but not "Groups", then is it interpreted as "What if User were not a member of any groupsSubjectAccessReviewStatusSubjectAccessReviewStatusallowed (boolean), requiredAllowed is required. True if the action would be allowed, false otherwise.denied (boolean)Denied is optional. True if the action would be denied, otherwise false. If both allowed is false and denied is false, then the authorizer has no opinion on whether to authorize the action. Denied may not be true if Allowed is true.evaluationError (string)EvaluationError is an indication that some error occurred during the authorization check. It is entirely possible to get an error and be able to continue determine authorization status in spite of it. For instance, RBAC can be missing a role, but enough roles are still present and bound to reason about the request.reason (string)Reason is optional. It indicates why a request was allowed or denied.Operationscreate create a SubjectAccessReviewHTTP RequestPOST /apis/authorization.k8s.io/v1/subjectaccessreviewsParametersbody: SubjectAccessReview, requireddryRun (in query): stringdryRunfieldManager (in query): stringfieldManagerfieldValidation (in query): stringfieldValidationpretty (in query): stringprettyResponse200 (SubjectAccessReview): OK201 (SubjectAccessReview): Created202 (SubjectAccessReview): Accepted401: UnauthorizedThis page is automatically generated.If you plan to report an issue with this page, mention that the page is auto-generated in your issue description. The fix may need to happen elsewhere in the Kubernetes project.

SubjectAccessReview


PodCertificateRequest v1alpha1PodCertificateRequest encodes a pod requesting a certificate from a given signer.apiVersion: certificates.k8s.io/v1alpha1import "k8s.io/api/certificates/v1alpha1"PodCertificateRequestPodCertificateRequest encodes a pod requesting a certificate from a given signer.Kubelets use this API to implement podCertificate projected volumesapiVersion: certificates.k8s.io/v1alpha1kind: PodCertificateRequestmetadata (ObjectMeta)metadata contains the object metadata.spec (PodCertificateRequestSpec), requiredspec contains the details about the certificate being requested.status (PodCertificateRequestStatus)status contains the issued certificate, and a standard set of conditions.PodCertificateRequestSpecPodCertificateRequestSpec describes the certificate request. All fields are immutable after creation.nodeName (string), requirednodeName is the name of the node the pod is assigned to.nodeUID (string), requirednodeUID is the UID of the node the pod is assigned to.pkixPublicKey ([]byte), requiredpkixPublicKey is the PKIX-serialized public key the signer will issue the certificate to.The key must be one of RSA3072, RSA4096, ECDSAP256, ECDSAP384, ECDSAP521, or ED25519. Note that this list may be expanded in the future.Signer implementations do not need to support all key types supported by kube-apiserver and kubelet. If a signer does not support the key type used for a given PodCertificateRequest, it must deny the request by setting a status.conditions entry with a type of "Denied" and a reason of "UnsupportedKeyType". It may also suggest a key type that it does support in the message field.podName (string), requiredpodName is the name of the pod into which the certificate will be mounted.podUID (string), requiredpodUID is the UID of the pod into which the certificate will be mounted.proofOfPossession ([]byte), requiredproofOfPossession proves that the requesting kubelet holds the private key corresponding to pkixPublicKey.It is contructed by signing the ASCII bytes of the pod's UID using pkixPublicKey.kube-apiserver validates the proof of possession during creation of the PodCertificateRequest.If the key is an RSA key, then the signature is over the ASCII bytes of the pod UID, using RSASSA-PSS from RFC 8017 (as implemented by the golang function crypto/rsa.SignPSS with nil options).If the key is an ECDSA key, then the signature is as described by SEC 1, Version 2.0 (as implemented by the golang library function crypto/ecdsa.SignASN1)If the key is an ED25519 key, the the signature is as described by the ED25519 Specification (as implemented by the golang library crypto/ed25519.Sign).serviceAccountName (string), requiredserviceAccountName is the name of the service account the pod is running as.serviceAccountUID (string), requiredserviceAccountUID is the UID of the service account the pod is running as.signerName (string), requiredsignerName indicates the requested signer.All signer names beginning with kubernetes.io are reserved for use by the Kubernetes project. There is currently one well-known signer documented by the Kubernetes project, kubernetes.io/kube-apiserver-client-pod, which will issue client certificates understood by kube-apiserver. It is currently unimplemented.maxExpirationSeconds (int32)maxExpirationSeconds is the maximum lifetime permitted for the certificate.If omitted, kube-apiserver will set it to 86400(24 hours). kube-apiserver will reject values shorter than 3600 (1 hour). The maximum allowable value is 7862400 (91 days).The signer implementation is then free to issue a certificate with any lifetime shorter than MaxExpirationSeconds, but no shorter than 3600 seconds (1 hour). This constraint is enforced by kube-apiserver. kubernetes.io signers will never issue certificates with a lifetime longer than 24 hours.PodCertificateRequestStatusPodCertificateRequestStatus describes the status of the request, and holds the certificate data if the request is issued.beginRefreshAt (Time)beginRefreshAt is the time at which the kubelet should begin trying to refresh the certificate. This field is set via the /status subresource, and must be set at the same time as certificateChain. Once populated, this field is immutable.This field is only a hint. Kubelet may start refreshing before or after this time if necessary.Time is a wrapper around time.Time which supports correct marshaling to YAML and JSON. Wrappers are provided for many of the factory methods that the time package offers.certificateChain (string)certificateChain is populated with an issued certificate by the signer. This field is set via the /status subresource. Once populated, this field is immutable.If the certificate signing request is denied, a condition of type "Denied" is added and this field remains empty. If the signer cannot issue the certificate, a condition of type "Failed" is added and this field remains empty.Validation requirements:certificateChain must consist of one or more PEM-formatted certificates.Each entry must be a valid PEM-wrapped, DER-encoded ASN.1 Certificate as described in section 4 of RFC5280.If more than one block is present, and the definition of the requested spec.signerName does not indicate otherwise, the first block is the issued certificate, and subsequent blocks should be treated as intermediate certificates and presented in TLS handshakes. When projecting the chain into a pod volume, kubelet will drop any data in-between the PEM blocks, as well as any PEM block headers.conditions ([]Condition)Patch strategy: merge on key typeMap: unique values on key type will be kept during a mergeconditions applied to the request.The types "Issued", "Denied", and "Failed" have special handling. At most one of these conditions may be present, and they must have status "True".If the request is denied with Reason=UnsupportedKeyType, the signer may suggest a key type that will work in the message field.Condition contains details for one aspect of the current state of this API Resource.conditions.lastTransitionTime (Time), requiredlastTransitionTime is the last time the condition transitioned from one status to another. This should be when the underlying condition changed. If that is not known, then using the time when the API field changed is acceptable.Time is a wrapper around time.Time which supports correct marshaling to YAML and JSON. Wrappers are provided for many of the factory methods that the time package offers.conditions.message (string), requiredmessage is a human readable message indicating details about the transition. This may be an empty string.conditions.reason (string), requiredreason contains a programmatic identifier indicating the reason for the condition's last transition. Producers of specific condition types may define expected values and meanings for this field, and whether the values are considered a guaranteed API. The value should be a CamelCase string. This field may not be empty.conditions.status (string), requiredstatus of the condition, one of True, False, Unknown.conditions.type (string), requiredtype of condition in CamelCase or in foo.example.com/CamelCase.conditions.observedGeneration (int64)observedGeneration represents the .metadata.generation that the condition was set based upon. For instance, if .metadata.generation is currently 12, but the .status.conditions[x].observedGeneration is 9, the condition is out of date with respect to the current state of the instance.notAfter (Time)notAfter is the time at which the certificate expires. The value must be the same as the notAfter value in the leaf certificate in certificateChain. This field is set via the /status subresource. Once populated, it is immutable. The signer must set this field at the same time it sets certificateChain.Time is a wrapper around time.Time which supports correct marshaling to YAML and JSON. Wrappers are provided for many of the factory methods that the time package offers.notBefore (Time)notBefore is the time at which the certificate becomes valid. The value must be the same as the notBefore value in the leaf certificate in certificateChain. This field is set via the /status subresource. Once populated, it is immutable. The signer must set this field at the same time it sets certificateChain.Time is a wrapper around time.Time which supports correct marshaling to YAML and JSON. Wrappers are provided for many of the factory methods that the time package offers.PodCertificateRequestListPodCertificateRequestList is a collection of PodCertificateRequest objectsapiVersion: certificates.k8s.io/v1alpha1kind: PodCertificateRequestListmetadata (ListMeta)metadata contains the list metadata.items ([]PodCertificateRequest), requireditems is a collection of PodCertificateRequest objectsOperationsget read the specified PodCertificateRequestHTTP RequestGET /apis/certificates.k8s.io/v1alpha1/namespaces/{namespace}/podcertificaterequests/{name}Parametersname (in path): string, requiredname of the PodCertificateRequestnamespace (in path): string, requirednamespacepretty (in query): stringprettyResponse200 (PodCertificateRequest): OK401: Unauthorizedget read status of the specified PodCertificateRequestHTTP RequestGET /apis/certificates.k8s.io/v1alpha1/namespaces/{namespace}/podcertificaterequests/{name}/statusParametersname (in path): string, requiredname of the PodCertificateRequestnamespace (in path): string, requirednamespacepretty (in query): stringprettyResponse200 (PodCertificateRequest): OK401: Unauthorizedlist list or watch objects of kind PodCertificateRequestHTTP RequestGET /apis/certificates.k8s.io/v1alpha1/namespaces/{namespace}/podcertificaterequestsParametersnamespace (in path): string, requirednamespaceallowWatchBookmarks (in query): booleanallowWatchBookmarkscontinue (in query): stringcontinuefieldSelector (in query): stringfieldSelectorlabelSelector (in query): stringlabelSelectorlimit (in query): integerlimitpretty (in query): stringprettyresourceVersion (in query): stringresourceVersionresourceVersionMatch (in query): stringresourceVersionMatchsendInitialEvents (in query): booleansendInitialEventstimeoutSeconds (in query): integertimeoutSecondswatch (in query): booleanwatchResponse200 (PodCertificateRequestList): OK401: Unauthorizedlist list or watch objects of kind PodCertificateRequestHTTP RequestGET /apis/certificates.k8s.io/v1alpha1/podcertificaterequestsParametersallowWatchBookmarks (in query): booleanallowWatchBookmarkscontinue (in query): stringcontinuefieldSelector (in query): stringfieldSelectorlabelSelector (in query): stringlabelSelectorlimit (in query): integerlimitpretty (in query): stringprettyresourceVersion (in query): stringresourceVersionresourceVersionMatch (in query): stringresourceVersionMatchsendInitialEvents (in query): booleansendInitialEventstimeoutSeconds (in query): integertimeoutSecondswatch (in query): booleanwatchResponse200 (PodCertificateRequestList): OK401: Unauthorizedcreate create a PodCertificateRequestHTTP RequestPOST /apis/certificates.k8s.io/v1alpha1/namespaces/{namespace}/podcertificaterequestsParametersnamespace (in path): string, requirednamespacebody: PodCertificateRequest, requireddryRun (in query): stringdryRunfieldManager (in query): stringfieldManagerfieldValidation (in query): stringfieldValidationpretty (in query): stringprettyResponse200 (PodCertificateRequest): OK201 (PodCertificateRequest): Created202 (PodCertificateRequest): Accepted401: Unauthorizedupdate replace the specified PodCertificateRequestHTTP RequestPUT /apis/certificates.k8s.io/v1alpha1/namespaces/{namespace}/podcertificaterequests/{name}Parametersname (in path): string, requiredname of the PodCertificateRequestnamespace (in path): string, requirednamespacebody: PodCertificateRequest, requireddryRun (in query): stringdryRunfieldManager (in query): stringfieldManagerfieldValidation (in query): stringfieldValidationpretty (in query): stringprettyResponse200 (PodCertificateRequest): OK201 (PodCertificateRequest): Created401: Unauthorizedupdate replace status of the specified PodCertificateRequestHTTP RequestPUT /apis/certificates.k8s.io/v1alpha1/namespaces/{namespace}/podcertificaterequests/{name}/statusParametersname (in path): string, requiredname of the PodCertificateRequestnamespace (in path): string, requirednamespacebody: PodCertificateRequest, requireddryRun (in query): stringdryRunfieldManager (in query): stringfieldManagerfieldValidation (in query): stringfieldValidationpretty (in query): stringprettyResponse200 (PodCertificateRequest): OK201 (PodCertificateRequest): Created401: Unauthorizedpatch partially update the specified PodCertificateRequestHTTP RequestPATCH /apis/certificates.k8s.io/v1alpha1/namespaces/{namespace}/podcertificaterequests/{name}Parametersname (in path): string, requiredname of the PodCertificateRequestnamespace (in path): string, requirednamespacebody: Patch, requireddryRun (in query): stringdryRunfieldManager (in query): stringfieldManagerfieldValidation (in query): stringfieldValidationforce (in query): booleanforcepretty (in query): stringprettyResponse200 (PodCertificateRequest): OK201 (PodCertificateRequest): Created401: Unauthorizedpatch partially update status of the specified PodCertificateRequestHTTP RequestPATCH /apis/certificates.k8s.io/v1alpha1/namespaces/{namespace}/podcertificaterequests/{name}/statusParametersname (in path): string, requiredname of the PodCertificateRequestnamespace (in path): string, requirednamespacebody: Patch, requireddryRun (in query): stringdryRunfieldManager (in query): stringfieldManagerfieldValidation (in query): stringfieldValidationforce (in query): booleanforcepretty (in query): stringprettyResponse200 (PodCertificateRequest): OK201 (PodCertificateRequest): Created401: Unauthorizeddelete delete a PodCertificateRequestHTTP RequestDELETE /apis/certificates.k8s.io/v1alpha1/namespaces/{namespace}/podcertificaterequests/{name}Parametersname (in path): string, requiredname of the PodCertificateRequestnamespace (in path): string, requirednamespacebody: DeleteOptionsdryRun (in query): stringdryRungracePeriodSeconds (in query): integergracePeriodSecondsignoreStoreReadErrorWithClusterBreakingPotential (in query): booleanignoreStoreReadErrorWithClusterBreakingPotentialpretty (in query): stringprettypropagationPolicy (in query): stringpropagationPolicyResponse200 (Status): OK202 (Status): Accepted401: Unauthorizeddeletecollection delete collection of PodCertificateRequestHTTP RequestDELETE /apis/certificates.k8s.io/v1alpha1/namespaces/{namespace}/podcertificaterequestsParametersnamespace (in path): string, requirednamespacebody: DeleteOptionscontinue (in query): stringcontinuedryRun (in query): stringdryRunfieldSelector (in query): stringfieldSelectorgracePeriodSeconds (in query): integergracePeriodSecondsignoreStoreReadErrorWithClusterBreakingPotential (in query): booleanignoreStoreReadErrorWithClusterBreakingPotentiallabelSelector (in query): stringlabelSelectorlimit (in query): integerlimitpretty (in query): stringprettypropagationPolicy (in query): stringpropagationPolicyresourceVersion (in query): stringresourceVersionresourceVersionMatch (in query): stringresourceVersionMatchsendInitialEvents (in query): booleansendInitialEventstimeoutSeconds (in query): integertimeoutSecondsResponse200 (Status): OK401: UnauthorizedThis page is automatically generated.If you plan to report an issue with this page, mention that the page is auto-generated in your issue description. The fix may need to happen elsewhere in the Kubernetes project.

PodCertificateRequest v1alpha1


Container RuntimesNote: Dockershim has been removed from the Kubernetes project as of release 1.24. Read the Dockershim Removal FAQ for further details.You need to install a container runtime into each node in the cluster so that Pods can run there. This page outlines what is involved and describes related tasks for setting up nodes.Kubernetes 1.34 requires that you use a runtime that conforms with the Container Runtime Interface (CRI).See CRI version support for more information.This page provides an outline of how to use several common container runtimes with Kubernetes.containerdCRI-ODocker EngineMirantis Container RuntimeNote:Kubernetes releases before v1.24 included a direct integration with Docker Engine, using a component named dockershim. That special direct integration is no longer part of Kubernetes (this removal was announced as part of the v1.20 release). You can read Check whether Dockershim removal affects you to understand how this removal might affect you. To learn about migrating from using dockershim, see Migrating from dockershim.If you are running a version of Kubernetes other than v1.34, check the documentation for that version.Install and configure prerequisitesNetwork configurationBy default, the Linux kernel does not allow IPv4 packets to be routed between interfaces. Most Kubernetes cluster networking implementations will change this setting (if needed), but some might expect the administrator to do it for them. (Some might also expect other sysctl parameters to be set, kernel modules to be loaded, etc; consult the documentation for your specific network implementation.)Enable IPv4 packet forwardingTo manually enable IPv4 packet forwarding:# sysctl params required by setup, params persist across reboots cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf net.ipv4.ip_forward = 1 EOF # Apply sysctl params without reboot sudo sysctl --system Verify that net.ipv4.ip_forward is set to 1 with:sysctl net.ipv4.ip_forward cgroup driversOn Linux, control groups are used to constrain resources that are allocated to processes.Both the kubelet and the underlying container runtime need to interface with control groups to enforce resource management for pods and containers and set resources such as cpu/memory requests and limits. To interface with control groups, the kubelet and the container runtime need to use a cgroup driver. It's critical that the kubelet and the container runtime use the same cgroup driver and are configured the same.There are two cgroup drivers available:cgroupfssystemdcgroupfs driverThe cgroupfs driver is the default cgroup driver in the kubelet. When the cgroupfs driver is used, the kubelet and the container runtime directly interface with the cgroup filesystem to configure cgroups.The cgroupfs driver is not recommended when systemd is the init system because systemd expects a single cgroup manager on the system. Additionally, if you use cgroup v2, use the systemd cgroup driver instead of cgroupfs.systemd cgroup driverWhen systemd is chosen as the init system for a Linux distribution, the init process generates and consumes a root control group (cgroup) and acts as a cgroup manager.systemd has a tight integration with cgroups and allocates a cgroup per systemd unit. As a result, if you use systemd as the init system with the cgroupfs driver, the system gets two different cgroup managers.Two cgroup managers result in two views of the available and in-use resources in the system. In some cases, nodes that are configured to use cgroupfs for the kubelet and container runtime, but use systemd for the rest of the processes become unstable under resource pressure.The approach to mitigate this instability is to use systemd as the cgroup driver for the kubelet and the container runtime when systemd is the selected init system.To set systemd as the cgroup driver, edit the KubeletConfiguration option of cgroupDriver and set it to systemd. For example:apiVersion: kubelet.config.k8s.io/v1beta1 kind: KubeletConfiguration ... cgroupDriver: systemd Note:Starting with v1.22 and later, when creating a cluster with kubeadm, if the user does not set the cgroupDriver field under KubeletConfiguration, kubeadm defaults it to systemd.If you configure systemd as the cgroup driver for the kubelet, you must also configure systemd as the cgroup driver for the container runtime. Refer to the documentation for your container runtime for instructions. For example:containerdCRI-OIn Kubernetes 1.34, with the KubeletCgroupDriverFromCRI feature gate enabled and a container runtime that supports the RuntimeConfig CRI RPC, the kubelet automatically detects the appropriate cgroup driver from the runtime, and ignores the cgroupDriver setting within the kubelet configuration.However, older versions of container runtimes (specifically, containerd 1.y and below) do not support the RuntimeConfig CRI RPC, and may not respond correctly to this query, and thus the Kubelet falls back to using the value in its own --cgroup-driver flag.In Kubernetes 1.36, this fallback behavior will be dropped, and older versions of containerd will fail with newer kubelets.Caution:Changing the cgroup driver of a Node that has joined a cluster is a sensitive operation. If the kubelet has created Pods using the semantics of one cgroup driver, changing the container runtime to another cgroup driver can cause errors when trying to re-create the Pod sandbox for such existing Pods. Restarting the kubelet may not solve such errors.If you have automation that makes it feasible, replace the node with another using the updated configuration, or reinstall it using automation.Migrating to the systemd driver in kubeadm managed clustersIf you wish to migrate to the systemd cgroup driver in existing kubeadm managed clusters, follow configuring a cgroup driver.CRI version supportYour container runtime must support at least v1alpha2 of the container runtime interface.Kubernetes starting v1.26 only works with v1 of the CRI API. Earlier versions default to v1 version, however if a container runtime does not support the v1 API, the kubelet falls back to using the (deprecated) v1alpha2 API instead.Container runtimesNote: This section links to third party projects that provide functionality required by Kubernetes. The Kubernetes project authors aren't responsible for these projects, which are listed alphabetically. To add a project to this list, read the content guide before submitting a change. More information.containerdThis section outlines the necessary steps to use containerd as CRI runtime.To install containerd on your system, follow the instructions on getting started with containerd. Return to this step once you've created a valid config.toml configuration file.You can find this file under the path /etc/containerd/config.toml.You can find this file under the path C:\Program Files\containerd\config.toml.On Linux the default CRI socket for containerd is /run/containerd/containerd.sock. On Windows the default CRI endpoint is npipe://./pipe/containerd-containerd.Configuring the systemd cgroup driverTo use the systemd cgroup driver in /etc/containerd/config.toml with runc, set the following config based on your Containerd versionContainerd versions 1.x:[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc] ... [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options] SystemdCgroup = true Containerd versions 2.x:[plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runc] ... [plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runc.options] SystemdCgroup = true The systemd cgroup driver is recommended if you use cgroup v2.Note:If you installed containerd from a package (for example, RPM or .deb), you may find that the CRI integration plugin is disabled by default.You need CRI support enabled to use containerd with Kubernetes. Make sure that cri is not included in thedisabled_plugins list within /etc/containerd/config.toml; if you made changes to that file, also restart containerd.If you experience container crash loops after the initial cluster installation or after installing a CNI, the containerd configuration provided with the package might contain incompatible configuration parameters. Consider resetting the containerd configuration with containerd config default > /etc/containerd/config.toml as specified in getting-started.md and then set the configuration parameters specified above accordingly.If you apply this change, make sure to restart containerd:sudo systemctl restart containerd When using kubeadm, manually configure the cgroup driver for kubelet.In Kubernetes v1.28, you can enable automatic detection of the cgroup driver as an alpha feature. See systemd cgroup driver for more details.Overriding the sandbox (pause) imageIn your containerd config you can overwrite the sandbox image by setting the following config:[plugins."io.containerd.grpc.v1.cri"] sandbox_image = "registry.k8s.io/pause:3.10" You might need to restart containerd as well once you've updated the config file: systemctl restart containerd.CRI-OThis section contains the necessary steps to install CRI-O as a container runtime.To install CRI-O, follow CRI-O Install Instructions.cgroup driverCRI-O uses the systemd cgroup driver per default, which is likely to work fine for you. To switch to the cgroupfs cgroup driver, either edit /etc/crio/crio.conf or place a drop-in configuration in /etc/crio/crio.conf.d/02-cgroup-manager.conf, for example:[crio.runtime] conmon_cgroup = "pod" cgroup_manager = "cgroupfs" You should also note the changed conmon_cgroup, which has to be set to the value pod when using CRI-O with cgroupfs. It is generally necessary to keep the cgroup driver configuration of the kubelet (usually done via kubeadm) and CRI-O in sync.In Kubernetes v1.28, you can enable automatic detection of the cgroup driver as an alpha feature. See systemd cgroup driver for more details.For CRI-O, the CRI socket is /var/run/crio/crio.sock by default.Overriding the sandbox (pause) imageIn your CRI-O config you can set the following config value:[crio.image] pause_image="registry.k8s.io/pause:3.10" This config option supports live configuration reload to apply this change: systemctl reload crio or by sending SIGHUP to the crio process.Docker EngineNote:These instructions assume that you are using the cri-dockerd adapter to integrate Docker Engine with Kubernetes.On each of your nodes, install Docker for your Linux distribution as per Install Docker Engine.Install cri-dockerd, following the directions in the install section of the documentation.For cri-dockerd, the CRI socket is /run/cri-dockerd.sock by default.Mirantis Container RuntimeMirantis Container Runtime (MCR) is a commercially available container runtime that was formerly known as Docker Enterprise Edition.You can use Mirantis Container Runtime with Kubernetes using the open source cri-dockerd component, included with MCR.To learn more about how to install Mirantis Container Runtime, visit MCR Deployment Guide.Check the systemd unit named cri-docker.socket to find out the path to the CRI socket.Overriding the sandbox (pause) imageThe cri-dockerd adapter accepts a command line argument for specifying which container image to use as the Pod infrastructure container (“pause image”). The command line argument to use is --pod-infra-container-image.What's nextAs well as a container runtime, your cluster will need a working network plugin.Items on this page refer to third party products or projects that provide functionality required by Kubernetes. The Kubernetes project authors aren't responsible for those third-party products or projects. See the CNCF website guidelines for more details.You should read the content guide before proposing a change that adds an extra third-party link.

Container Runtimes


Volume Snapshot ClassesThis document describes the concept of VolumeSnapshotClass in Kubernetes. Familiarity with volume snapshots and storage classes is suggested.IntroductionJust like StorageClass provides a way for administrators to describe the "classes" of storage they offer when provisioning a volume, VolumeSnapshotClass provides a way to describe the "classes" of storage when provisioning a volume snapshot.The VolumeSnapshotClass ResourceEach VolumeSnapshotClass contains the fields driver, deletionPolicy, and parameters, which are used when a VolumeSnapshot belonging to the class needs to be dynamically provisioned.The name of a VolumeSnapshotClass object is significant, and is how users can request a particular class. Administrators set the name and other parameters of a class when first creating VolumeSnapshotClass objects, and the objects cannot be updated once they are created.Note:Installation of the CRDs is the responsibility of the Kubernetes distribution. Without the required CRDs present, the creation of a VolumeSnapshotClass fails.apiVersion: snapshot.storage.k8s.io/v1 kind: VolumeSnapshotClass metadata: name: csi-hostpath-snapclass driver: hostpath.csi.k8s.io deletionPolicy: Delete parameters: Administrators can specify a default VolumeSnapshotClass for VolumeSnapshots that don't request any particular class to bind to by adding the snapshot.storage.kubernetes.io/is-default-class: "true" annotation:apiVersion: snapshot.storage.k8s.io/v1 kind: VolumeSnapshotClass metadata: name: csi-hostpath-snapclass annotations: snapshot.storage.kubernetes.io/is-default-class: "true" driver: hostpath.csi.k8s.io deletionPolicy: Delete parameters: If multiple CSI drivers exist, a default VolumeSnapshotClass can be specified for each of them.VolumeSnapshotClass dependenciesWhen you create a VolumeSnapshot without specifying a VolumeSnapshotClass, Kubernetes automatically selects a default VolumeSnapshotClass that has a CSI driver matching the CSI driver of the PVC’s StorageClass.This behavior allows multiple default VolumeSnapshotClass objects to coexist in a cluster, as long as each one is associated with a unique CSI driver.Always ensure that there is only one default VolumeSnapshotClass for each CSI driver. If multiple default VolumeSnapshotClass objects are created using the same CSI driver, a VolumeSnapshot creation will fail because Kubernetes cannot determine which one to use.DriverVolume snapshot classes have a driver that determines what CSI volume plugin is used for provisioning VolumeSnapshots. This field must be specified.DeletionPolicyVolume snapshot classes have a deletionPolicy. It enables you to configure what happens to a VolumeSnapshotContent when the VolumeSnapshot object it is bound to is to be deleted. The deletionPolicy of a volume snapshot class can either be Retain or Delete. This field must be specified.If the deletionPolicy is Delete, then the underlying storage snapshot will be deleted along with the VolumeSnapshotContent object. If the deletionPolicy is Retain, then both the underlying snapshot and VolumeSnapshotContent remain.ParametersVolume snapshot classes have parameters that describe volume snapshots belonging to the volume snapshot class. Different parameters may be accepted depending on the driver.

Volume Snapshot Classes


Securing a ClusterThis document covers topics related to protecting a cluster from accidental or malicious access and provides recommendations on overall security.Before you beginYou need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. It is recommended to run this tutorial on a cluster with at least two nodes that are not acting as control plane hosts. If you do not already have a cluster, you can create one by using minikube or you can use one of these Kubernetes playgrounds:iximiuz LabsKillercodaKodeKloudPlay with KubernetesTo check the version, enter kubectl version.Controlling access to the Kubernetes APIAs Kubernetes is entirely API-driven, controlling and limiting who can access the cluster and what actions they are allowed to perform is the first line of defense.Use Transport Layer Security (TLS) for all API trafficKubernetes expects that all API communication in the cluster is encrypted by default with TLS, and the majority of installation methods will allow the necessary certificates to be created and distributed to the cluster components. Note that some components and installation methods may enable local ports over HTTP and administrators should familiarize themselves with the settings of each component to identify potentially unsecured traffic.API AuthenticationChoose an authentication mechanism for the API servers to use that matches the common access patterns when you install a cluster. For instance, small, single-user clusters may wish to use a simple certificate or static Bearer token approach. Larger clusters may wish to integrate an existing OIDC or LDAP server that allow users to be subdivided into groups.All API clients must be authenticated, even those that are part of the infrastructure like nodes, proxies, the scheduler, and volume plugins. These clients are typically service accounts or use x509 client certificates, and they are created automatically at cluster startup or are setup as part of the cluster installation.Consult the authentication reference document for more information.API AuthorizationOnce authenticated, every API call is also expected to pass an authorization check. Kubernetes ships an integrated Role-Based Access Control (RBAC) component that matches an incoming user or group to a set of permissions bundled into roles. These permissions combine verbs (get, create, delete) with resources (pods, services, nodes) and can be namespace-scoped or cluster-scoped. A set of out-of-the-box roles are provided that offer reasonable default separation of responsibility depending on what actions a client might want to perform. It is recommended that you use the Node and RBAC authorizers together, in combination with the NodeRestriction admission plugin.As with authentication, simple and broad roles may be appropriate for smaller clusters, but as more users interact with the cluster, it may become necessary to separate teams into separate namespaces with more limited roles.With authorization, it is important to understand how updates on one object may cause actions in other places. For instance, a user may not be able to create pods directly, but allowing them to create a deployment, which creates pods on their behalf, will let them create those pods indirectly. Likewise, deleting a node from the API will result in the pods scheduled to that node being terminated and recreated on other nodes. The out-of-the box roles represent a balance between flexibility and common use cases, but more limited roles should be carefully reviewed to prevent accidental escalation. You can make roles specific to your use case if the out-of-box ones don't meet your needs.Consult the authorization reference section for more information.Controlling access to the KubeletKubelets expose HTTPS endpoints which grant powerful control over the node and containers. By default Kubelets allow unauthenticated access to this API.Production clusters should enable Kubelet authentication and authorization.Consult the Kubelet authentication/authorization reference for more information.Controlling the capabilities of a workload or user at runtimeAuthorization in Kubernetes is intentionally high level, focused on coarse actions on resources. More powerful controls exist as policies to limit by use case how those objects act on the cluster, themselves, and other resources.Limiting resource usage on a clusterResource quota limits the number or capacity of resources granted to a namespace. This is most often used to limit the amount of CPU, memory, or persistent disk a namespace can allocate, but can also control how many pods, services, or volumes exist in each namespace.Limit ranges restrict the maximum or minimum size of some of the resources above, to prevent users from requesting unreasonably high or low values for commonly reserved resources like memory, or to provide default limits when none are specified.Controlling what privileges containers run withA pod definition contains a security context that allows it to request access to run as a specific Linux user on a node (like root), access to run privileged or access the host network, and other controls that would otherwise allow it to run unfettered on a hosting node.You can configure Pod security admission to enforce use of a particular Pod Security Standard in a namespace, or to detect breaches.Generally, most application workloads need limited access to host resources so they can successfully run as a root process (uid 0) without access to host information. However, considering the privileges associated with the root user, you should write application containers to run as a non-root user. Similarly, administrators who wish to prevent client applications from escaping their containers should apply the Baseline or Restricted Pod Security Standard.Preventing containers from loading unwanted kernel modulesThe Linux kernel automatically loads kernel modules from disk if needed in certain circumstances, such as when a piece of hardware is attached or a filesystem is mounted. Of particular relevance to Kubernetes, even unprivileged processes can cause certain network-protocol-related kernel modules to be loaded, just by creating a socket of the appropriate type. This may allow an attacker to exploit a security hole in a kernel module that the administrator assumed was not in use.To prevent specific modules from being automatically loaded, you can uninstall them from the node, or add rules to block them. On most Linux distributions, you can do that by creating a file such as /etc/modprobe.d/kubernetes-blacklist.conf with contents like:# DCCP is unlikely to be needed, has had multiple serious # vulnerabilities, and is not well-maintained. blacklist dccp # SCTP is not used in most Kubernetes clusters, and has also had # vulnerabilities in the past. blacklist sctp To block module loading more generically, you can use a Linux Security Module (such as SELinux) to completely deny the module_request permission to containers, preventing the kernel from loading modules for containers under any circumstances. (Pods would still be able to use modules that had been loaded manually, or modules that were loaded by the kernel on behalf of some more-privileged process.)Restricting network accessThe network policies for a namespace allows application authors to restrict which pods in other namespaces may access pods and ports within their namespaces. Many of the supported Kubernetes networking providers now respect network policy.Quota and limit ranges can also be used to control whether users may request node ports or load-balanced services, which on many clusters can control whether those users applications are visible outside of the cluster.Additional protections may be available that control network rules on a per-plugin or per- environment basis, such as per-node firewalls, physically separating cluster nodes to prevent cross talk, or advanced networking policy.Restricting cloud metadata API accessCloud platforms (AWS, Azure, GCE, etc.) often expose metadata services locally to instances. By default these APIs are accessible by pods running on an instance and can contain cloud credentials for that node, or provisioning data such as kubelet credentials. These credentials can be used to escalate within the cluster or to other cloud services under the same account.When running Kubernetes on a cloud platform, limit permissions given to instance credentials, use network policies to restrict pod access to the metadata API, and avoid using provisioning data to deliver secrets.Controlling which nodes pods may accessBy default, there are no restrictions on which nodes may run a pod. Kubernetes offers a rich set of policies for controlling placement of pods onto nodes and the taint-based pod placement and eviction that are available to end users. For many clusters use of these policies to separate workloads can be a convention that authors adopt or enforce via tooling.As an administrator, a beta admission plugin PodNodeSelector can be used to force pods within a namespace to default or require a specific node selector, and if end users cannot alter namespaces, this can strongly limit the placement of all of the pods in a specific workload.Protecting cluster components from compromiseThis section describes some common patterns for protecting clusters from compromise.Restrict access to etcdWrite access to the etcd backend for the API is equivalent to gaining root on the entire cluster, and read access can be used to escalate fairly quickly. Administrators should always use strong credentials from the API servers to their etcd server, such as mutual auth via TLS client certificates, and it is often recommended to isolate the etcd servers behind a firewall that only the API servers may access.Caution:Allowing other components within the cluster to access the master etcd instance with read or write access to the full keyspace is equivalent to granting cluster-admin access. Using separate etcd instances for non-master components or using etcd ACLs to restrict read and write access to a subset of the keyspace is strongly recommended.Enable audit loggingThe audit logger is a beta feature that records actions taken by the API for later analysis in the event of a compromise. It is recommended to enable audit logging and archive the audit file on a secure server.Restrict access to alpha or beta featuresAlpha and beta Kubernetes features are in active development and may have limitations or bugs that result in security vulnerabilities. Always assess the value an alpha or beta feature may provide against the possible risk to your security posture. When in doubt, disable features you do not use.Rotate infrastructure credentials frequentlyThe shorter the lifetime of a secret or credential the harder it is for an attacker to make use of that credential. Set short lifetimes on certificates and automate their rotation. Use an authentication provider that can control how long issued tokens are available and use short lifetimes where possible. If you use service-account tokens in external integrations, plan to rotate those tokens frequently. For example, once the bootstrap phase is complete, a bootstrap token used for setting up nodes should be revoked or its authorization removed.Review third party integrations before enabling themMany third party integrations to Kubernetes may alter the security profile of your cluster. When enabling an integration, always review the permissions that an extension requests before granting it access. For example, many security integrations may request access to view all secrets on your cluster which is effectively making that component a cluster admin. When in doubt, restrict the integration to functioning in a single namespace if possible.Components that create pods may also be unexpectedly powerful if they can do so inside namespaces like the kube-system namespace, because those pods can gain access to service account secrets or run with elevated permissions if those service accounts are granted access to permissive PodSecurityPolicies.If you use Pod Security admission and allow any component to create Pods within a namespace that permits privileged Pods, those Pods may be able to escape their containers and use this widened access to elevate their privileges.You should not allow untrusted components to create Pods in any system namespace (those with names that start with kube-) nor in any namespace where that access grant allows the possibility of privilege escalation.Encrypt secrets at restIn general, the etcd database will contain any information accessible via the Kubernetes API and may grant an attacker significant visibility into the state of your cluster. Always encrypt your backups using a well reviewed backup and encryption solution, and consider using full disk encryption where possible.Kubernetes supports optional encryption at rest for information in the Kubernetes API. This lets you ensure that when Kubernetes stores data for objects (for example, Secret or ConfigMap objects), the API server writes an encrypted representation of the object. That encryption means that even someone who has access to etcd backup data is unable to view the content of those objects. In Kubernetes 1.34 you can also encrypt custom resources; encryption-at-rest for extension APIs defined in CustomResourceDefinitions was added to Kubernetes as part of the v1.26 release.Receiving alerts for security updates and reporting vulnerabilitiesJoin the kubernetes-announce group for emails about security announcements. See the security reporting page for more on how to report vulnerabilities.What's nextSecurity Checklist for additional information on Kubernetes security guidance.Seccomp Node Reference

Securing a Cluster


Troubleshooting kubeadmAs with any program, you might run into an error installing or running kubeadm. This page lists some common failure scenarios and have provided steps that can help you understand and fix the problem.If your problem is not listed below, please follow the following steps:If you think your problem is a bug with kubeadm:Go to github.com/kubernetes/kubeadm and search for existing issues.If no issue exists, please open one and follow the issue template.If you are unsure about how kubeadm works, you can ask on Slack in #kubeadm, or open a question on StackOverflow. Please include relevant tags like #kubernetes and #kubeadm so folks can help you.Not possible to join a v1.18 Node to a v1.17 cluster due to missing RBACIn v1.18 kubeadm added prevention for joining a Node in the cluster if a Node with the same name already exists. This required adding RBAC for the bootstrap-token user to be able to GET a Node object.However this causes an issue where kubeadm join from v1.18 cannot join a cluster created by kubeadm v1.17.To workaround the issue you have two options:Execute kubeadm init phase bootstrap-token on a control-plane node using kubeadm v1.18. Note that this enables the rest of the bootstrap-token permissions as well.orApply the following RBAC manually using kubectl apply -f ...:apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: kubeadm:get-nodes rules: - apiGroups: - "" resources: - nodes verbs: - get --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: kubeadm:get-nodes roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: kubeadm:get-nodes subjects: - apiGroup: rbac.authorization.k8s.io kind: Group name: system:bootstrappers:kubeadm:default-node-token ebtables or some similar executable not found during installationIf you see the following warnings while running kubeadm init[preflight] WARNING: ebtables not found in system path [preflight] WARNING: ethtool not found in system path Then you may be missing ebtables, ethtool or a similar executable on your node. You can install them with the following commands:For Ubuntu/Debian users, run apt install ebtables ethtool.For CentOS/Fedora users, run yum install ebtables ethtool.kubeadm blocks waiting for control plane during installationIf you notice that kubeadm init hangs after printing out the following line:[apiclient] Created API client, waiting for the control plane to become ready This may be caused by a number of problems. The most common are:network connection problems. Check that your machine has full network connectivity before continuing.the cgroup driver of the container runtime differs from that of the kubelet. To understand how to configure it properly, see Configuring a cgroup driver.control plane containers are crashlooping or hanging. You can check this by running docker ps and investigating each container by running docker logs. For other container runtime, see Debugging Kubernetes nodes with crictl.kubeadm blocks when removing managed containersThe following could happen if the container runtime halts and does not remove any Kubernetes-managed containers:sudo kubeadm reset [preflight] Running pre-flight checks [reset] Stopping the kubelet service [reset] Unmounting mounted directories in "/var/lib/kubelet" [reset] Removing kubernetes-managed containers (block) A possible solution is to restart the container runtime and then re-run kubeadm reset. You can also use crictl to debug the state of the container runtime. See Debugging Kubernetes nodes with crictl.Pods in RunContainerError, CrashLoopBackOff or Error stateRight after kubeadm init there should not be any pods in these states.If there are pods in one of these states right after kubeadm init, please open an issue in the kubeadm repo. coredns (or kube-dns) should be in the Pending state until you have deployed the network add-on.If you see Pods in the RunContainerError, CrashLoopBackOff or Error state after deploying the network add-on and nothing happens to coredns (or kube-dns), it's very likely that the Pod Network add-on that you installed is somehow broken. You might have to grant it more RBAC privileges or use a newer version. Please file an issue in the Pod Network providers' issue tracker and get the issue triaged there.coredns is stuck in the Pending stateThis is expected and part of the design. kubeadm is network provider-agnostic, so the admin should install the pod network add-on of choice. You have to install a Pod Network before CoreDNS may be deployed fully. Hence the Pending state before the network is set up.HostPort services do not workThe HostPort and HostIP functionality is available depending on your Pod Network provider. Please contact the author of the Pod Network add-on to find out whether HostPort and HostIP functionality are available.Calico, Canal, and Flannel CNI providers are verified to support HostPort.For more information, see the CNI portmap documentation.If your network provider does not support the portmap CNI plugin, you may need to use the NodePort feature of services or use HostNetwork=true.Pods are not accessible via their Service IPMany network add-ons do not yet enable hairpin mode which allows pods to access themselves via their Service IP. This is an issue related to CNI. Please contact the network add-on provider to get the latest status of their support for hairpin mode.If you are using VirtualBox (directly or via Vagrant), you will need to ensure that hostname -i returns a routable IP address. By default, the first interface is connected to a non-routable host-only network. A work around is to modify /etc/hosts, see this Vagrantfile for an example.TLS certificate errorsThe following error indicates a possible certificate mismatch.# kubectl get pods Unable to connect to the server: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes") Verify that the $HOME/.kube/config file contains a valid certificate, and regenerate a certificate if necessary. The certificates in a kubeconfig file are base64 encoded. The base64 --decode command can be used to decode the certificate and openssl x509 -text -noout can be used for viewing the certificate information.Unset the KUBECONFIG environment variable using:unset KUBECONFIG Or set it to the default KUBECONFIG location:export KUBECONFIG=/etc/kubernetes/admin.conf Another workaround is to overwrite the existing kubeconfig for the "admin" user:mv $HOME/.kube $HOME/.kube.bak mkdir $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config Kubelet client certificate rotation failsBy default, kubeadm configures a kubelet with automatic rotation of client certificates by using the /var/lib/kubelet/pki/kubelet-client-current.pem symlink specified in /etc/kubernetes/kubelet.conf. If this rotation process fails you might see errors such as x509: certificate has expired or is not yet valid in kube-apiserver logs. To fix the issue you must follow these steps:Backup and delete /etc/kubernetes/kubelet.conf and /var/lib/kubelet/pki/kubelet-client* from the failed node.From a working control plane node in the cluster that has /etc/kubernetes/pki/ca.key execute kubeadm kubeconfig user --org system:nodes --client-name system:node:$NODE > kubelet.conf. $NODE must be set to the name of the existing failed node in the cluster. Modify the resulted kubelet.conf manually to adjust the cluster name and server endpoint, or pass kubeconfig user --config (see Generating kubeconfig files for additional users). If your cluster does not have the ca.key you must sign the embedded certificates in the kubelet.conf externally.Copy this resulted kubelet.conf to /etc/kubernetes/kubelet.conf on the failed node.Restart the kubelet (systemctl restart kubelet) on the failed node and wait for /var/lib/kubelet/pki/kubelet-client-current.pem to be recreated.Manually edit the kubelet.conf to point to the rotated kubelet client certificates, by replacing client-certificate-data and client-key-data with:client-certificate: /var/lib/kubelet/pki/kubelet-client-current.pem client-key: /var/lib/kubelet/pki/kubelet-client-current.pem Restart the kubelet.Make sure the node becomes Ready.Default NIC When using flannel as the pod network in VagrantThe following error might indicate that something was wrong in the pod network:Error from server (NotFound): the server could not find the requested resource If you're using flannel as the pod network inside Vagrant, then you will have to specify the default interface name for flannel.Vagrant typically assigns two interfaces to all VMs. The first, for which all hosts are assigned the IP address 10.0.2.15, is for external traffic that gets NATed.This may lead to problems with flannel, which defaults to the first interface on a host. This leads to all hosts thinking they have the same public IP address. To prevent this, pass the --iface eth1 flag to flannel so that the second interface is chosen.Non-public IP used for containersIn some situations kubectl logs and kubectl run commands may return with the following errors in an otherwise functional cluster:Error from server: Get https://10.19.0.41:10250/containerLogs/default/mysql-ddc65b868-glc5m/mysql: dial tcp 10.19.0.41:10250: getsockopt: no route to host This may be due to Kubernetes using an IP that can not communicate with other IPs on the seemingly same subnet, possibly by policy of the machine provider.DigitalOcean assigns a public IP to eth0 as well as a private one to be used internally as anchor for their floating IP feature, yet kubelet will pick the latter as the node's InternalIP instead of the public one.Use ip addr show to check for this scenario instead of ifconfig because ifconfig will not display the offending alias IP address. Alternatively an API endpoint specific to DigitalOcean allows to query for the anchor IP from the droplet:curl http://169.254.169.254/metadata/v1/interfaces/public/0/anchor_ipv4/address The workaround is to tell kubelet which IP to use using --node-ip. When using DigitalOcean, it can be the public one (assigned to eth0) or the private one (assigned to eth1) should you want to use the optional private network. The kubeletExtraArgs section of the kubeadm NodeRegistrationOptions structure can be used for this.Then restart kubelet:systemctl daemon-reload systemctl restart kubelet coredns pods have CrashLoopBackOff or Error stateIf you have nodes that are running SELinux with an older version of Docker, you might experience a scenario where the coredns pods are not starting. To solve that, you can try one of the following options:Upgrade to a newer version of Docker.Disable SELinux.Modify the coredns deployment to set allowPrivilegeEscalation to true:kubectl -n kube-system get deployment coredns -o yaml | \ sed 's/allowPrivilegeEscalation: false/allowPrivilegeEscalation: true/g' | \ kubectl apply -f - Another cause for CoreDNS to have CrashLoopBackOff is when a CoreDNS Pod deployed in Kubernetes detects a loop. A number of workarounds are available to avoid Kubernetes trying to restart the CoreDNS Pod every time CoreDNS detects the loop and exits.Warning:Disabling SELinux or setting allowPrivilegeEscalation to true can compromise the security of your cluster.etcd pods restart continuallyIf you encounter the following error:rpc error: code = 2 desc = oci runtime error: exec failed: container_linux.go:247: starting container process caused "process_linux.go:110: decoding init error from pipe caused \"read parent: connection reset by peer\"" This issue appears if you run CentOS 7 with Docker 1.13.1.84. This version of Docker can prevent the kubelet from executing into the etcd container.To work around the issue, choose one of these options:Roll back to an earlier version of Docker, such as 1.13.1-75yum downgrade docker-1.13.1-75.git8633870.el7.centos.x86_64 docker-client-1.13.1-75.git8633870.el7.centos.x86_64 docker-common-1.13.1-75.git8633870.el7.centos.x86_64 Install one of the more recent recommended versions, such as 18.06:sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo yum install docker-ce-18.06.1.ce-3.el7.x86_64 Not possible to pass a comma separated list of values to arguments inside a --component-extra-args flagkubeadm init flags such as --component-extra-args allow you to pass custom arguments to a control-plane component like the kube-apiserver. However, this mechanism is limited due to the underlying type used for parsing the values (mapStringString).If you decide to pass an argument that supports multiple, comma-separated values such as --apiserver-extra-args "enable-admission-plugins=LimitRanger,NamespaceExists" this flag will fail with flag: malformed pair, expect string=string. This happens because the list of arguments for --apiserver-extra-args expects key=value pairs and in this case NamespacesExists is considered as a key that is missing a value.Alternatively, you can try separating the key=value pairs like so: --apiserver-extra-args "enable-admission-plugins=LimitRanger,enable-admission-plugins=NamespaceExists" but this will result in the key enable-admission-plugins only having the value of NamespaceExists.A known workaround is to use the kubeadm configuration file.kube-proxy scheduled before node is initialized by cloud-controller-managerIn cloud provider scenarios, kube-proxy can end up being scheduled on new worker nodes before the cloud-controller-manager has initialized the node addresses. This causes kube-proxy to fail to pick up the node's IP address properly and has knock-on effects to the proxy function managing load balancers.The following error can be seen in kube-proxy Pods:server.go:610] Failed to retrieve node IP: host IP unknown; known addresses: [] proxier.go:340] invalid nodeIP, initializing kube-proxy with 127.0.0.1 as nodeIP A known solution is to patch the kube-proxy DaemonSet to allow scheduling it on control-plane nodes regardless of their conditions, keeping it off of other nodes until their initial guarding conditions abate:kubectl -n kube-system patch ds kube-proxy -p='{ "spec": { "template": { "spec": { "tolerations": [ { "key": "CriticalAddonsOnly", "operator": "Exists" }, { "effect": "NoSchedule", "key": "node-role.kubernetes.io/control-plane" } ] } } } }' The tracking issue for this problem is here./usr is mounted read-only on nodesOn Linux distributions such as Fedora CoreOS or Flatcar Container Linux, the directory /usr is mounted as a read-only filesystem. For flex-volume support, Kubernetes components like the kubelet and kube-controller-manager use the default path of /usr/libexec/kubernetes/kubelet-plugins/volume/exec/, yet the flex-volume directory must be writeable for the feature to work.Note:FlexVolume was deprecated in the Kubernetes v1.23 release.To workaround this issue, you can configure the flex-volume directory using the kubeadm configuration file.On the primary control-plane Node (created using kubeadm init), pass the following file using --config:apiVersion: kubeadm.k8s.io/v1beta4 kind: InitConfiguration nodeRegistration: kubeletExtraArgs: - name: "volume-plugin-dir" value: "/opt/libexec/kubernetes/kubelet-plugins/volume/exec/" --- apiVersion: kubeadm.k8s.io/v1beta4 kind: ClusterConfiguration controllerManager: extraArgs: - name: "flex-volume-plugin-dir" value: "/opt/libexec/kubernetes/kubelet-plugins/volume/exec/" On joining Nodes:apiVersion: kubeadm.k8s.io/v1beta4 kind: JoinConfiguration nodeRegistration: kubeletExtraArgs: - name: "volume-plugin-dir" value: "/opt/libexec/kubernetes/kubelet-plugins/volume/exec/" Alternatively, you can modify /etc/fstab to make the /usr mount writeable, but please be advised that this is modifying a design principle of the Linux distribution.kubeadm upgrade plan prints out context deadline exceeded error messageThis error message is shown when upgrading a Kubernetes cluster with kubeadm in the case of running an external etcd. This is not a critical bug and happens because older versions of kubeadm perform a version check on the external etcd cluster. You can proceed with kubeadm upgrade apply ....This issue is fixed as of version 1.19.kubeadm reset unmounts /var/lib/kubeletIf /var/lib/kubelet is being mounted, performing a kubeadm reset will effectively unmount it.To workaround the issue, re-mount the /var/lib/kubelet directory after performing the kubeadm reset operation.This is a regression introduced in kubeadm 1.15. The issue is fixed in 1.20.Cannot use the metrics-server securely in a kubeadm clusterIn a kubeadm cluster, the metrics-server can be used insecurely by passing the --kubelet-insecure-tls to it. This is not recommended for production clusters.If you want to use TLS between the metrics-server and the kubelet there is a problem, since kubeadm deploys a self-signed serving certificate for the kubelet. This can cause the following errors on the side of the metrics-server:x509: certificate signed by unknown authority x509: certificate is valid for IP-foo not IP-bar See Enabling signed kubelet serving certificates to understand how to configure the kubelets in a kubeadm cluster to have properly signed serving certificates.Also see How to run the metrics-server securely.Upgrade fails due to etcd hash not changingOnly applicable to upgrading a control plane node with a kubeadm binary v1.28.3 or later, where the node is currently managed by kubeadm versions v1.28.0, v1.28.1 or v1.28.2.Here is the error message you may encounter:[upgrade/etcd] Failed to upgrade etcd: couldn't upgrade control plane. kubeadm has tried to recover everything into the earlier state. Errors faced: static Pod hash for component etcd on Node kinder-upgrade-control-plane-1 did not change after 5m0s: timed out waiting for the condition [upgrade/etcd] Waiting for previous etcd to become available I0907 10:10:09.109104 3704 etcd.go:588] [etcd] attempting to see if all cluster endpoints ([https://172.17.0.6:2379/ https://172.17.0.4:2379/ https://172.17.0.3:2379/]) are available 1/10 [upgrade/etcd] Etcd was rolled back and is now available static Pod hash for component etcd on Node kinder-upgrade-control-plane-1 did not change after 5m0s: timed out waiting for the condition couldn't upgrade control plane. kubeadm has tried to recover everything into the earlier state. Errors faced k8s.io/kubernetes/cmd/kubeadm/app/phases/upgrade.rollbackOldManifests cmd/kubeadm/app/phases/upgrade/staticpods.go:525 k8s.io/kubernetes/cmd/kubeadm/app/phases/upgrade.upgradeComponent cmd/kubeadm/app/phases/upgrade/staticpods.go:254 k8s.io/kubernetes/cmd/kubeadm/app/phases/upgrade.performEtcdStaticPodUpgrade cmd/kubeadm/app/phases/upgrade/staticpods.go:338 ... The reason for this failure is that the affected versions generate an etcd manifest file with unwanted defaults in the PodSpec. This will result in a diff from the manifest comparison, and kubeadm will expect a change in the Pod hash, but the kubelet will never update the hash.There are two way to workaround this issue if you see it in your cluster:The etcd upgrade can be skipped between the affected versions and v1.28.3 (or later) by using:kubeadm upgrade {apply|node} [version] --etcd-upgrade=false This is not recommended in case a new etcd version was introduced by a later v1.28 patch version.Before upgrade, patch the manifest for the etcd static pod, to remove the problematic defaulted attributes:diff --git a/etc/kubernetes/manifests/etcd_defaults.yaml b/etc/kubernetes/manifests/etcd_origin.yaml index d807ccbe0aa..46b35f00e15 100644 --- a/etc/kubernetes/manifests/etcd_defaults.yaml +++ b/etc/kubernetes/manifests/etcd_origin.yaml @@ -43,7 +43,6 @@ spec: scheme: HTTP initialDelaySeconds: 10 periodSeconds: 10 - successThreshold: 1 timeoutSeconds: 15 name: etcd resources: @@ -59,26 +58,18 @@ spec: scheme: HTTP initialDelaySeconds: 10 periodSeconds: 10 - successThreshold: 1 timeoutSeconds: 15 - terminationMessagePath: /dev/termination-log - terminationMessagePolicy: File volumeMounts: - mountPath: /var/lib/etcd name: etcd-data - mountPath: /etc/kubernetes/pki/etcd name: etcd-certs - dnsPolicy: ClusterFirst - enableServiceLinks: true hostNetwork: true priority: 2000001000 priorityClassName: system-node-critical - restartPolicy: Always - schedulerName: default-scheduler securityContext: seccompProfile: type: RuntimeDefault - terminationGracePeriodSeconds: 30 volumes: - hostPath: path: /etc/kubernetes/pki/etcd More information can be found in the tracking issue for this bug.

Troubleshooting kubeadm


Manual Rotation of CA CertificatesThis page shows how to manually rotate the certificate authority (CA) certificates.Before you beginYou need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. It is recommended to run this tutorial on a cluster with at least two nodes that are not acting as control plane hosts. If you do not already have a cluster, you can create one by using minikube or you can use one of these Kubernetes playgrounds:iximiuz LabsKillercodaKodeKloudPlay with KubernetesFor more information about authentication in Kubernetes, see Authenticating.For more information about best practices for CA certificates, see Single root CA.Rotate the CA certificates manuallyCaution:Make sure to back up your certificate directory along with configuration files and any other necessary files.This approach assumes operation of the Kubernetes control plane in a HA configuration with multiple API servers. Graceful termination of the API server is also assumed so clients can cleanly disconnect from one API server and reconnect to another.Configurations with a single API server will experience unavailability while the API server is being restarted.Distribute the new CA certificates and private keys (for example: ca.crt, ca.key, front-proxy-ca.crt, and front-proxy-ca.key) to all your control plane nodes in the Kubernetes certificates directory.Update the --root-ca-file flag for the kube-controller-manager to include both old and new CA, then restart the kube-controller-manager.Any ServiceAccount created after this point will get Secrets that include both old and new CAs.Note:The files specified by the kube-controller-manager flags --client-ca-file and --cluster-signing-cert-file cannot be CA bundles. If these flags and --root-ca-file point to the same ca.crt file which is now a bundle (includes both old and new CA) you will face an error. To workaround this problem you can copy the new CA to a separate file and make the flags --client-ca-file and --cluster-signing-cert-file point to the copy. Once ca.crt is no longer a bundle you can restore the problem flags to point to ca.crt and delete the copy.Issue 1350 for kubeadm tracks an bug with the kube-controller-manager being unable to accept a CA bundle.Wait for the controller manager to update ca.crt in the service account Secrets to include both old and new CA certificates.If any Pods are started before new CA is used by API servers, the new Pods get this update and will trust both old and new CAs.Restart all pods using in-cluster configurations (for example: kube-proxy, CoreDNS, etc) so they can use the updated certificate authority data from Secrets that link to ServiceAccounts.Make sure CoreDNS, kube-proxy and other Pods using in-cluster configurations are working as expected.Append the both old and new CA to the file against --client-ca-file and --kubelet-certificate-authority flag in the kube-apiserver configuration.Append the both old and new CA to the file against --client-ca-file flag in the kube-scheduler configuration.Update certificates for user accounts by replacing the content of client-certificate-data and client-key-data respectively.For information about creating certificates for individual user accounts, see Configure certificates for user accounts.Additionally, update the certificate-authority-data section in the kubeconfig files, respectively with Base64-encoded old and new certificate authority dataUpdate the --root-ca-file flag for the Cloud Controller Manager to include both old and new CA, then restart the cloud-controller-manager.Note:If your cluster does not have a cloud-controller-manager, you can skip this step.Follow the steps below in a rolling fashion.Restart any other aggregated API servers or webhook handlers to trust the new CA certificates.Restart the kubelet by update the file against clientCAFile in kubelet configuration and certificate-authority-data in kubelet.conf to use both the old and new CA on all nodes.If your kubelet is not using client certificate rotation, update client-certificate-data and client-key-data in kubelet.conf on all nodes along with the kubelet client certificate file usually found in /var/lib/kubelet/pki.Restart API servers with the certificates (apiserver.crt, apiserver-kubelet-client.crt and front-proxy-client.crt) signed by new CA. You can use the existing private keys or new private keys. If you changed the private keys then update these in the Kubernetes certificates directory as well.Since the Pods in your cluster trust both old and new CAs, there will be a momentarily disconnection after which pods' Kubernetes clients reconnect to the new API server. The new API server uses a certificate signed by the new CA.Restart the kube-scheduler to use and trust the new CAs.Make sure control plane components logs no TLS errors.Note: To generate certificates and private keys for your cluster using the `openssl` command line tool, see [Certificates (`openssl`)](/docs/tasks/administer-cluster/certificates/#openssl). You can also use [`cfssl`](/docs/tasks/administer-cluster/certificates/#cfssl). Annotate any DaemonSets and Deployments to trigger pod replacement in a safer rolling fashion.for namespace in $(kubectl get namespace -o jsonpath='{.items[*].metadata.name}'); do for name in $(kubectl get deployments -n $namespace -o jsonpath='{.items[*].metadata.name}'); do kubectl patch deployment -n ${namespace} ${name} -p '{"spec":{"template":{"metadata":{"annotations":{"ca-rotation": "1"}}}}}'; done for name in $(kubectl get daemonset -n $namespace -o jsonpath='{.items[*].metadata.name}'); do kubectl patch daemonset -n ${namespace} ${name} -p '{"spec":{"template":{"metadata":{"annotations":{"ca-rotation": "1"}}}}}'; done done Note: To limit the number of concurrent disruptions that your application experiences, see [configure pod disruption budget](/docs/tasks/run-application/configure-pdb/). Depending on how you use StatefulSets you may also need to perform similar rolling replacement. If your cluster is using bootstrap tokens to join nodes, update the ConfigMap cluster-info in the kube-public namespace with new CA.base64_encoded_ca="$(base64 -w0 /etc/kubernetes/pki/ca.crt)" kubectl get cm/cluster-info --namespace kube-public -o yaml | \ /bin/sed "s/\(certificate-authority-data:\).*/\1 ${base64_encoded_ca}/" | \ kubectl apply -f - Verify the cluster functionality.Check the logs from control plane components, along with the kubelet and the kube-proxy. Ensure those components are not reporting any TLS errors; see looking at the logs for more details.Validate logs from any aggregated api servers and pods using in-cluster config.Once the cluster functionality is successfully verified:Update all service account tokens to include new CA certificate only.All pods using an in-cluster kubeconfig will eventually need to be restarted to pick up the new Secret, so that no Pods are relying on the old cluster CA.Restart the control plane components by removing the old CA from the kubeconfig files and the files against --client-ca-file, --root-ca-file flags resp.On each node, restart the kubelet by removing the old CA from file against the clientCAFile flag and from the kubelet kubeconfig file. You should carry this out as a rolling update.If your cluster lets you make this change, you can also roll it out by replacing nodes rather than reconfiguring them.

Manual Rotation of CA Certificates


kubectl rollout pauseSynopsisMark the provided resource as paused.Paused resources will not be reconciled by a controller. Use "kubectl rollout resume" to resume a paused resource. Currently only deployments support being paused.kubectl rollout pause RESOURCE Examples # Mark the nginx deployment as paused # Any current state of the deployment will continue its function; new updates # to the deployment will not have an effect as long as the deployment is paused kubectl rollout pause deployment/nginx Options--allow-missing-template-keys Default: trueIf true, ignore any errors in templates when a field or map key is missing in the template. Only applies to golang and jsonpath output formats.--field-manager string Default: "kubectl-rollout"Name of the manager used to track field ownership.-f, --filename stringsFilename, directory, or URL to files identifying the resource to get from a server.-h, --helphelp for pause-k, --kustomize stringProcess the kustomization directory. This flag can't be used together with -f or -R.-o, --output stringOutput format. One of: (json, yaml, name, go-template, go-template-file, template, templatefile, jsonpath, jsonpath-as-json, jsonpath-file).-R, --recursiveProcess the directory used in -f, --filename recursively. Useful when you want to manage related manifests organized within the same directory.-l, --selector stringSelector (label query) to filter on, supports '=', '==', '!=', 'in', 'notin'.(e.g. -l key1=value1,key2=value2,key3 in (value3)). Matching objects must satisfy all of the specified label constraints.--show-managed-fieldsIf true, keep the managedFields when printing objects in JSON or YAML format.--template stringTemplate string or path to template file to use when -o=go-template, -o=go-template-file. The template format is golang templates [http://golang.org/pkg/text/template/#pkg-overview].Parent Options Inherited--as stringUsername to impersonate for the operation. User could be a regular user or a service account in a namespace.--as-group stringsGroup to impersonate for the operation, this flag can be repeated to specify multiple groups.--as-uid stringUID to impersonate for the operation.--cache-dir string Default: "$HOME/.kube/cache"Default cache directory--certificate-authority stringPath to a cert file for the certificate authority--client-certificate stringPath to a client certificate file for TLS--client-key stringPath to a client key file for TLS--cluster stringThe name of the kubeconfig cluster to use--context stringThe name of the kubeconfig context to use--disable-compressionIf true, opt-out of response compression for all requests to the server--insecure-skip-tls-verifyIf true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure--kubeconfig stringPath to the kubeconfig file to use for CLI requests.--kuberc stringPath to the kuberc file to use for preferences. This can be disabled by exporting KUBECTL_KUBERC=false feature gate or turning off the feature KUBERC=off.--match-server-versionRequire server version to match client version-n, --namespace stringIf present, the namespace scope for this CLI request--password stringPassword for basic authentication to the API server--profile string Default: "none"Name of profile to capture. One of (none|cpu|heap|goroutine|threadcreate|block|mutex)--profile-output string Default: "profile.pprof"Name of the file to write the profile to--request-timeout string Default: "0"The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests.-s, --server stringThe address and port of the Kubernetes API server--storage-driver-buffer-duration duration Default: 1m0sWrites in the storage driver will be buffered for this duration, and committed to the non memory backends as a single transaction--storage-driver-db string Default: "cadvisor"database name--storage-driver-host string Default: "localhost:8086"database host:port--storage-driver-password string Default: "root"database password--storage-driver-secureuse secure connection with database--storage-driver-table string Default: "stats"table name--storage-driver-user string Default: "root"database username--tls-server-name stringServer name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used--token stringBearer token for authentication to the API server--user stringThe name of the kubeconfig user to use--username stringUsername for basic authentication to the API server--version version[=true]--version, --version=raw prints version information and quits; --version=vX.Y.Z... sets the reported version--warnings-as-errorsTreat warnings received from the server as errors and exit with a non-zero exit codeSee Alsokubectl rollout - Manage the rollout of a resourceThis page is automatically generated.If you plan to report an issue with this page, mention that the page is auto-generated in your issue description. The fix may need to happen elsewhere in the Kubernetes project.

kubectl rollout pause