kubectl rollout statusSynopsisShow the status of the rollout.By default 'rollout status' will watch the status of the latest rollout until it's done. If you don't want to wait for the rollout to finish then you can use --watch=false. Note that if a new rollout starts in-between, then 'rollout status' will continue watching the latest revision. If you want to pin to a specific revision and abort if it is rolled over by another revision, use --revision=N where N is the revision you need to watch for.kubectl rollout status (TYPE NAME | TYPE/NAME) [flags] Examples # Watch the rollout status of a deployment kubectl rollout status deployment/nginx Options-f, --filename stringsFilename, directory, or URL to files identifying the resource to get from a server.-h, --helphelp for status-k, --kustomize stringProcess the kustomization directory. This flag can't be used together with -f or -R.-R, --recursiveProcess the directory used in -f, --filename recursively. Useful when you want to manage related manifests organized within the same directory.--revision intPin to a specific revision for showing its status. Defaults to 0 (last revision).-l, --selector stringSelector (label query) to filter on, supports '=', '==', '!=', 'in', 'notin'.(e.g. -l key1=value1,key2=value2,key3 in (value3)). Matching objects must satisfy all of the specified label constraints.--timeout durationThe length of time to wait before ending watch, zero means never. Any other values should contain a corresponding time unit (e.g. 1s, 2m, 3h).-w, --watch Default: trueWatch the status of the rollout until it's done.Parent Options Inherited--as stringUsername to impersonate for the operation. User could be a regular user or a service account in a namespace.--as-group stringsGroup to impersonate for the operation, this flag can be repeated to specify multiple groups.--as-uid stringUID to impersonate for the operation.--cache-dir string Default: "$HOME/.kube/cache"Default cache directory--certificate-authority stringPath to a cert file for the certificate authority--client-certificate stringPath to a client certificate file for TLS--client-key stringPath to a client key file for TLS--cluster stringThe name of the kubeconfig cluster to use--context stringThe name of the kubeconfig context to use--disable-compressionIf true, opt-out of response compression for all requests to the server--insecure-skip-tls-verifyIf true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure--kubeconfig stringPath to the kubeconfig file to use for CLI requests.--kuberc stringPath to the kuberc file to use for preferences. This can be disabled by exporting KUBECTL_KUBERC=false feature gate or turning off the feature KUBERC=off.--match-server-versionRequire server version to match client version-n, --namespace stringIf present, the namespace scope for this CLI request--password stringPassword for basic authentication to the API server--profile string Default: "none"Name of profile to capture. One of (none|cpu|heap|goroutine|threadcreate|block|mutex)--profile-output string Default: "profile.pprof"Name of the file to write the profile to--request-timeout string Default: "0"The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests.-s, --server stringThe address and port of the Kubernetes API server--storage-driver-buffer-duration duration Default: 1m0sWrites in the storage driver will be buffered for this duration, and committed to the non memory backends as a single transaction--storage-driver-db string Default: "cadvisor"database name--storage-driver-host string Default: "localhost:8086"database host:port--storage-driver-password string Default: "root"database password--storage-driver-secureuse secure connection with database--storage-driver-table string Default: "stats"table name--storage-driver-user string Default: "root"database username--tls-server-name stringServer name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used--token stringBearer token for authentication to the API server--user stringThe name of the kubeconfig user to use--username stringUsername for basic authentication to the API server--version version[=true]--version, --version=raw prints version information and quits; --version=vX.Y.Z... sets the reported version--warnings-as-errorsTreat warnings received from the server as errors and exit with a non-zero exit codeSee Alsokubectl rollout - Manage the rollout of a resourceThis page is automatically generated.If you plan to report an issue with this page, mention that the page is auto-generated in your issue description. The fix may need to happen elsewhere in the Kubernetes project.

kubectl rollout status


EndpointSliceEndpointSlice represents a set of service endpoints.apiVersion: discovery.k8s.io/v1import "k8s.io/api/discovery/v1"EndpointSliceEndpointSlice represents a set of service endpoints. Most EndpointSlices are created by the EndpointSlice controller to represent the Pods selected by Service objects. For a given service there may be multiple EndpointSlice objects which must be joined to produce the full set of endpoints; you can find all of the slices for a given service by listing EndpointSlices in the service's namespace whose kubernetes.io/service-name label contains the service's name.apiVersion: discovery.k8s.io/v1kind: EndpointSlicemetadata (ObjectMeta)Standard object's metadata.addressType (string), requiredaddressType specifies the type of address carried by this EndpointSlice. All addresses in this slice must be the same type. This field is immutable after creation. The following address types are currently supported: * IPv4: Represents an IPv4 Address. * IPv6: Represents an IPv6 Address. * FQDN: Represents a Fully Qualified Domain Name. (Deprecated) The EndpointSlice controller only generates, and kube-proxy only processes, slices of addressType "IPv4" and "IPv6". No semantics are defined for the "FQDN" type.endpoints ([]Endpoint), requiredAtomic: will be replaced during a mergeendpoints is a list of unique endpoints in this slice. Each slice may include a maximum of 1000 endpoints.Endpoint represents a single logical "backend" implementing a service.endpoints.addresses ([]string), requiredSet: unique values will be kept during a mergeaddresses of this endpoint. For EndpointSlices of addressType "IPv4" or "IPv6", the values are IP addresses in canonical form. The syntax and semantics of other addressType values are not defined. This must contain at least one address but no more than 100. EndpointSlices generated by the EndpointSlice controller will always have exactly 1 address. No semantics are defined for additional addresses beyond the first, and kube-proxy does not look at them.endpoints.conditions (EndpointConditions)conditions contains information about the current status of the endpoint.EndpointConditions represents the current condition of an endpoint.endpoints.conditions.ready (boolean)ready indicates that this endpoint is ready to receive traffic, according to whatever system is managing the endpoint. A nil value should be interpreted as "true". In general, an endpoint should be marked ready if it is serving and not terminating, though this can be overridden in some cases, such as when the associated Service has set the publishNotReadyAddresses flag.endpoints.conditions.serving (boolean)serving indicates that this endpoint is able to receive traffic, according to whatever system is managing the endpoint. For endpoints backed by pods, the EndpointSlice controller will mark the endpoint as serving if the pod's Ready condition is True. A nil value should be interpreted as "true".endpoints.conditions.terminating (boolean)terminating indicates that this endpoint is terminating. A nil value should be interpreted as "false".endpoints.deprecatedTopology (map[string]string)deprecatedTopology contains topology information part of the v1beta1 API. This field is deprecated, and will be removed when the v1beta1 API is removed (no sooner than kubernetes v1.24). While this field can hold values, it is not writable through the v1 API, and any attempts to write to it will be silently ignored. Topology information can be found in the zone and nodeName fields instead.endpoints.hints (EndpointHints)hints contains information associated with how an endpoint should be consumed.EndpointHints provides hints describing how an endpoint should be consumed.endpoints.hints.forNodes ([]ForNode)Atomic: will be replaced during a mergeforNodes indicates the node(s) this endpoint should be consumed by when using topology aware routing. May contain a maximum of 8 entries. This is an Alpha feature and is only used when the PreferSameTrafficDistribution feature gate is enabled.ForNode provides information about which nodes should consume this endpoint.endpoints.hints.forNodes.name (string), requiredname represents the name of the node.endpoints.hints.forZones ([]ForZone)Atomic: will be replaced during a mergeforZones indicates the zone(s) this endpoint should be consumed by when using topology aware routing. May contain a maximum of 8 entries.ForZone provides information about which zones should consume this endpoint.endpoints.hints.forZones.name (string), requiredname represents the name of the zone.endpoints.hostname (string)hostname of this endpoint. This field may be used by consumers of endpoints to distinguish endpoints from each other (e.g. in DNS names). Multiple endpoints which use the same hostname should be considered fungible (e.g. multiple A values in DNS). Must be lowercase and pass DNS Label (RFC 1123) validation.endpoints.nodeName (string)nodeName represents the name of the Node hosting this endpoint. This can be used to determine endpoints local to a Node.endpoints.targetRef (ObjectReference)targetRef is a reference to a Kubernetes object that represents this endpoint.endpoints.zone (string)zone is the name of the Zone this endpoint exists in.ports ([]EndpointPort)Atomic: will be replaced during a mergeports specifies the list of network ports exposed by each endpoint in this slice. Each port must have a unique name. Each slice may include a maximum of 100 ports. Services always have at least 1 port, so EndpointSlices generated by the EndpointSlice controller will likewise always have at least 1 port. EndpointSlices used for other purposes may have an empty ports list.EndpointPort represents a Port used by an EndpointSliceports.port (int32)port represents the port number of the endpoint. If the EndpointSlice is derived from a Kubernetes service, this must be set to the service's target port. EndpointSlices used for other purposes may have a nil port.ports.protocol (string)protocol represents the IP protocol for this port. Must be UDP, TCP, or SCTP. Default is TCP.ports.name (string)name represents the name of this port. All ports in an EndpointSlice must have a unique name. If the EndpointSlice is derived from a Kubernetes service, this corresponds to the Service.ports[].name. Name must either be an empty string or pass DNS_LABEL validation: * must be no more than 63 characters long. * must consist of lower case alphanumeric characters or '-'. * must start and end with an alphanumeric character. Default is empty string.ports.appProtocol (string)The application protocol for this port. This is used as a hint for implementations to offer richer behavior for protocols that they understand. This field follows standard Kubernetes label syntax. Valid values are either:Un-prefixed protocol names - reserved for IANA standard service names (as per RFC-6335 and https://www.iana.org/assignments/service-names).Kubernetes-defined prefixed names:'kubernetes.io/h2c' - HTTP/2 prior knowledge over cleartext as described in https://www.rfc-editor.org/rfc/rfc9113.html#name-starting-http-2-with-prior-'kubernetes.io/ws' - WebSocket over cleartext as described in https://www.rfc-editor.org/rfc/rfc6455'kubernetes.io/wss' - WebSocket over TLS as described in https://www.rfc-editor.org/rfc/rfc6455Other protocols should use implementation-defined prefixed names such as mycompany.com/my-custom-protocol.EndpointSliceListEndpointSliceList represents a list of endpoint slicesapiVersion: discovery.k8s.io/v1kind: EndpointSliceListmetadata (ListMeta)Standard list metadata.items ([]EndpointSlice), requireditems is the list of endpoint slicesOperationsget read the specified EndpointSliceHTTP RequestGET /apis/discovery.k8s.io/v1/namespaces/{namespace}/endpointslices/{name}Parametersname (in path): string, requiredname of the EndpointSlicenamespace (in path): string, requirednamespacepretty (in query): stringprettyResponse200 (EndpointSlice): OK401: Unauthorizedlist list or watch objects of kind EndpointSliceHTTP RequestGET /apis/discovery.k8s.io/v1/namespaces/{namespace}/endpointslicesParametersnamespace (in path): string, requirednamespaceallowWatchBookmarks (in query): booleanallowWatchBookmarkscontinue (in query): stringcontinuefieldSelector (in query): stringfieldSelectorlabelSelector (in query): stringlabelSelectorlimit (in query): integerlimitpretty (in query): stringprettyresourceVersion (in query): stringresourceVersionresourceVersionMatch (in query): stringresourceVersionMatchsendInitialEvents (in query): booleansendInitialEventstimeoutSeconds (in query): integertimeoutSecondswatch (in query): booleanwatchResponse200 (EndpointSliceList): OK401: Unauthorizedlist list or watch objects of kind EndpointSliceHTTP RequestGET /apis/discovery.k8s.io/v1/endpointslicesParametersallowWatchBookmarks (in query): booleanallowWatchBookmarkscontinue (in query): stringcontinuefieldSelector (in query): stringfieldSelectorlabelSelector (in query): stringlabelSelectorlimit (in query): integerlimitpretty (in query): stringprettyresourceVersion (in query): stringresourceVersionresourceVersionMatch (in query): stringresourceVersionMatchsendInitialEvents (in query): booleansendInitialEventstimeoutSeconds (in query): integertimeoutSecondswatch (in query): booleanwatchResponse200 (EndpointSliceList): OK401: Unauthorizedcreate create an EndpointSliceHTTP RequestPOST /apis/discovery.k8s.io/v1/namespaces/{namespace}/endpointslicesParametersnamespace (in path): string, requirednamespacebody: EndpointSlice, requireddryRun (in query): stringdryRunfieldManager (in query): stringfieldManagerfieldValidation (in query): stringfieldValidationpretty (in query): stringprettyResponse200 (EndpointSlice): OK201 (EndpointSlice): Created202 (EndpointSlice): Accepted401: Unauthorizedupdate replace the specified EndpointSliceHTTP RequestPUT /apis/discovery.k8s.io/v1/namespaces/{namespace}/endpointslices/{name}Parametersname (in path): string, requiredname of the EndpointSlicenamespace (in path): string, requirednamespacebody: EndpointSlice, requireddryRun (in query): stringdryRunfieldManager (in query): stringfieldManagerfieldValidation (in query): stringfieldValidationpretty (in query): stringprettyResponse200 (EndpointSlice): OK201 (EndpointSlice): Created401: Unauthorizedpatch partially update the specified EndpointSliceHTTP RequestPATCH /apis/discovery.k8s.io/v1/namespaces/{namespace}/endpointslices/{name}Parametersname (in path): string, requiredname of the EndpointSlicenamespace (in path): string, requirednamespacebody: Patch, requireddryRun (in query): stringdryRunfieldManager (in query): stringfieldManagerfieldValidation (in query): stringfieldValidationforce (in query): booleanforcepretty (in query): stringprettyResponse200 (EndpointSlice): OK201 (EndpointSlice): Created401: Unauthorizeddelete delete an EndpointSliceHTTP RequestDELETE /apis/discovery.k8s.io/v1/namespaces/{namespace}/endpointslices/{name}Parametersname (in path): string, requiredname of the EndpointSlicenamespace (in path): string, requirednamespacebody: DeleteOptionsdryRun (in query): stringdryRungracePeriodSeconds (in query): integergracePeriodSecondsignoreStoreReadErrorWithClusterBreakingPotential (in query): booleanignoreStoreReadErrorWithClusterBreakingPotentialpretty (in query): stringprettypropagationPolicy (in query): stringpropagationPolicyResponse200 (Status): OK202 (Status): Accepted401: Unauthorizeddeletecollection delete collection of EndpointSliceHTTP RequestDELETE /apis/discovery.k8s.io/v1/namespaces/{namespace}/endpointslicesParametersnamespace (in path): string, requirednamespacebody: DeleteOptionscontinue (in query): stringcontinuedryRun (in query): stringdryRunfieldSelector (in query): stringfieldSelectorgracePeriodSeconds (in query): integergracePeriodSecondsignoreStoreReadErrorWithClusterBreakingPotential (in query): booleanignoreStoreReadErrorWithClusterBreakingPotentiallabelSelector (in query): stringlabelSelectorlimit (in query): integerlimitpretty (in query): stringprettypropagationPolicy (in query): stringpropagationPolicyresourceVersion (in query): stringresourceVersionresourceVersionMatch (in query): stringresourceVersionMatchsendInitialEvents (in query): booleansendInitialEventstimeoutSeconds (in query): integertimeoutSecondsResponse200 (Status): OK401: UnauthorizedThis page is automatically generated.If you plan to report an issue with this page, mention that the page is auto-generated in your issue description. The fix may need to happen elsewhere in the Kubernetes project.

EndpointSlice


Security ChecklistBaseline checklist for ensuring security in Kubernetes clusters.This checklist aims at providing a basic list of guidance with links to more comprehensive documentation on each topic. It does not claim to be exhaustive and is meant to evolve.On how to read and use this document:The order of topics does not reflect an order of priority.Some checklist items are detailed in the paragraph below the list of each section.Caution:Checklists are not sufficient for attaining a good security posture on their own. A good security posture requires constant attention and improvement, but a checklist can be the first step on the never-ending journey towards security preparedness. Some of the recommendations in this checklist may be too restrictive or too lax for your specific security needs. Since Kubernetes security is not "one size fits all", each category of checklist items should be evaluated on its merits.Authentication & Authorization system:masters group is not used for user or component authentication after bootstrapping. The kube-controller-manager is running with --use-service-account-credentials enabled. The root certificate is protected (either an offline CA, or a managed online CA with effective access controls). Intermediate and leaf certificates have an expiry date no more than 3 years in the future. A process exists for periodic access review, and reviews occur no more than 24 months apart. The Role Based Access Control Good Practices are followed for guidance related to authentication and authorization.After bootstrapping, neither users nor components should authenticate to the Kubernetes API as system:masters. Similarly, running all of kube-controller-manager as system:masters should be avoided. In fact, system:masters should only be used as a break-glass mechanism, as opposed to an admin user.Network security CNI plugins in use support network policies. Ingress and egress network policies are applied to all workloads in the cluster. Default network policies within each namespace, selecting all pods, denying everything, are in place. If appropriate, a service mesh is used to encrypt all communications inside of the cluster. The Kubernetes API, kubelet API and etcd are not exposed publicly on Internet. Access from the workloads to the cloud metadata API is filtered. Use of LoadBalancer and ExternalIPs is restricted.A number of Container Network Interface (CNI) plugins plugins provide the functionality to restrict network resources that pods may communicate with. This is most commonly done through Network Policies which provide a namespaced resource to define rules. Default network policies that block all egress and ingress, in each namespace, selecting all pods, can be useful to adopt an allow list approach to ensure that no workloads are missed.Not all CNI plugins provide encryption in transit. If the chosen plugin lacks this feature, an alternative solution could be to use a service mesh to provide that functionality.The etcd datastore of the control plane should have controls to limit access and not be publicly exposed on the Internet. Furthermore, mutual TLS (mTLS) should be used to communicate securely with it. The certificate authority for this should be unique to etcd.External Internet access to the Kubernetes API server should be restricted to not expose the API publicly. Be careful, as many managed Kubernetes distributions are publicly exposing the API server by default. You can then use a bastion host to access the server.The kubelet API access should be restricted and not exposed publicly, the default authentication and authorization settings, when no configuration file specified with the --config flag, are overly permissive.If a cloud provider is used for hosting Kubernetes, the access from pods to the cloud metadata API 169.254.169.254 should also be restricted or blocked if not needed because it may leak information.For restricted LoadBalancer and ExternalIPs use, see CVE-2020-8554: Man in the middle using LoadBalancer or ExternalIPs and the DenyServiceExternalIPs admission controller for further information.Pod security RBAC rights to create, update, patch, delete workloads is only granted if necessary. Appropriate Pod Security Standards policy is applied for all namespaces and enforced. Memory limit is set for the workloads with a limit equal or inferior to the request. CPU limit might be set on sensitive workloads. For nodes that support it, Seccomp is enabled with appropriate syscalls profile for programs. For nodes that support it, AppArmor or SELinux is enabled with appropriate profile for programs.RBAC authorization is crucial but cannot be granular enough to have authorization on the Pods' resources (or on any resource that manages Pods). The only granularity is the API verbs on the resource itself, for example, create on Pods. Without additional admission, the authorization to create these resources allows direct unrestricted access to the schedulable nodes of a cluster.The Pod Security Standards define three different policies, privileged, baseline and restricted that limit how fields can be set in the PodSpec regarding security. These standards can be enforced at the namespace level with the new Pod Security admission, enabled by default, or by third-party admission webhook. Please note that, contrary to the removed PodSecurityPolicy admission it replaces, Pod Security admission can be easily combined with admission webhooks and external services.Pod Security admission restricted policy, the most restrictive policy of the Pod Security Standards set, can operate in several modes, warn, audit or enforce to gradually apply the most appropriate security context according to security best practices. Nevertheless, pods' security context should be separately investigated to limit the privileges and access pods may have on top of the predefined security standards, for specific use cases.For a hands-on tutorial on Pod Security, see the blog post Kubernetes 1.23: Pod Security Graduates to Beta.Memory and CPU limits should be set in order to restrict the memory and CPU resources a pod can consume on a node, and therefore prevent potential DoS attacks from malicious or breached workloads. Such policy can be enforced by an admission controller. Please note that CPU limits will throttle usage and thus can have unintended effects on auto-scaling features or efficiency i.e. running the process in best effort with the CPU resource available.Caution:Memory limit superior to request can expose the whole node to OOM issues.Enabling SeccompSeccomp stands for secure computing mode and has been a feature of the Linux kernel since version 2.6.12. It can be used to sandbox the privileges of a process, restricting the calls it is able to make from userspace into the kernel. Kubernetes lets you automatically apply seccomp profiles loaded onto a node to your Pods and containers.Seccomp can improve the security of your workloads by reducing the Linux kernel syscall attack surface available inside containers. The seccomp filter mode leverages BPF to create an allow or deny list of specific syscalls, named profiles.Since Kubernetes 1.27, you can enable the use of RuntimeDefault as the default seccomp profile for all workloads. A security tutorial is available on this topic. In addition, the Kubernetes Security Profiles Operator is a project that facilitates the management and use of seccomp in clusters.Note:Seccomp is only available on Linux nodes.Enabling AppArmor or SELinuxAppArmorAppArmor is a Linux kernel security module that can provide an easy way to implement Mandatory Access Control (MAC) and better auditing through system logs. A default AppArmor profile is enforced on nodes that support it, or a custom profile can be configured. Like seccomp, AppArmor is also configured through profiles, where each profile is either running in enforcing mode, which blocks access to disallowed resources or complain mode, which only reports violations. AppArmor profiles are enforced on a per-container basis, with an annotation, allowing for processes to gain just the right privileges.Note:AppArmor is only available on Linux nodes, and enabled in some Linux distributions.SELinuxSELinux is also a Linux kernel security module that can provide a mechanism for supporting access control security policies, including Mandatory Access Controls (MAC). SELinux labels can be assigned to containers or pods via their securityContext section.Note:SELinux is only available on Linux nodes, and enabled in some Linux distributions.Logs and auditing Audit logs, if enabled, are protected from general access.Pod placement Pod placement is done in accordance with the tiers of sensitivity of the application. Sensitive applications are running isolated on nodes or with specific sandboxed runtimes.Pods that are on different tiers of sensitivity, for example, an application pod and the Kubernetes API server, should be deployed onto separate nodes. The purpose of node isolation is to prevent an application container breakout to directly providing access to applications with higher level of sensitivity to easily pivot within the cluster. This separation should be enforced to prevent pods accidentally being deployed onto the same node. This could be enforced with the following features:Node SelectorsKey-value pairs, as part of the pod specification, that specify which nodes to deploy onto. These can be enforced at the namespace and cluster level with the PodNodeSelector admission controller.PodTolerationRestrictionAn admission controller that allows administrators to restrict permitted tolerations within a namespace. Pods within a namespace may only utilize the tolerations specified on the namespace object annotation keys that provide a set of default and allowed tolerations.RuntimeClassRuntimeClass is a feature for selecting the container runtime configuration. The container runtime configuration is used to run a Pod's containers and can provide more or less isolation from the host at the cost of performance overhead.Secrets ConfigMaps are not used to hold confidential data. Encryption at rest is configured for the Secret API. If appropriate, a mechanism to inject secrets stored in third-party storage is deployed and available. Service account tokens are not mounted in pods that don't require them. Bound service account token volume is in-use instead of non-expiring tokens.Secrets required for pods should be stored within Kubernetes Secrets as opposed to alternatives such as ConfigMap. Secret resources stored within etcd should be encrypted at rest.Pods needing secrets should have these automatically mounted through volumes, preferably stored in memory like with the emptyDir.medium option. Mechanism can be used to also inject secrets from third-party storages as volume, like the Secrets Store CSI Driver. This should be done preferentially as compared to providing the pods service account RBAC access to secrets. This would allow adding secrets into the pod as environment variables or files. Please note that the environment variable method might be more prone to leakage due to crash dumps in logs and the non-confidential nature of environment variable in Linux, as opposed to the permission mechanism on files.Service account tokens should not be mounted into pods that do not require them. This can be configured by setting automountServiceAccountToken to false either within the service account to apply throughout the namespace or specifically for a pod. For Kubernetes v1.22 and above, use Bound Service Accounts for time-bound service account credentials.Images Minimize unnecessary content in container images. Container images are configured to be run as unprivileged user. References to container images are made by sha256 digests (rather than tags) or the provenance of the image is validated by verifying the image's digital signature at deploy time via admission control. Container images are regularly scanned during creation and in deployment, and known vulnerable software is patched.Container image should contain the bare minimum to run the program they package. Preferably, only the program and its dependencies, building the image from the minimal possible base. In particular, image used in production should not contain shells or debugging utilities, as an ephemeral debug container can be used for troubleshooting.Build images to directly start with an unprivileged user by using the USER instruction in Dockerfile. The Security Context allows a container image to be started with a specific user and group with runAsUser and runAsGroup, even if not specified in the image manifest. However, the file permissions in the image layers might make it impossible to just start the process with a new unprivileged user without image modification.Avoid using image tags to reference an image, especially the latest tag, the image behind a tag can be easily modified in a registry. Prefer using the complete sha256 digest which is unique to the image manifest. This policy can be enforced via an ImagePolicyWebhook. Image signatures can also be automatically verified with an admission controller at deploy time to validate their authenticity and integrity.Scanning a container image can prevent critical vulnerabilities from being deployed to the cluster alongside the container image. Image scanning should be completed before deploying a container image to a cluster and is usually done as part of the deployment process in a CI/CD pipeline. The purpose of an image scan is to obtain information about possible vulnerabilities and their prevention in the container image, such as a Common Vulnerability Scoring System (CVSS) score. If the result of the image scans is combined with the pipeline compliance rules, only properly patched container images will end up in Production.Admission controllers An appropriate selection of admission controllers is enabled. A pod security policy is enforced by the Pod Security Admission or/and a webhook admission controller. The admission chain plugins and webhooks are securely configured.Admission controllers can help improve the security of the cluster. However, they can present risks themselves as they extend the API server and should be properly secured.The following lists present a number of admission controllers that could be considered to enhance the security posture of your cluster and application. It includes controllers that may be referenced in other parts of this document.This first group of admission controllers includes plugins enabled by default, consider to leave them enabled unless you know what you are doing:CertificateApprovalPerforms additional authorization checks to ensure the approving user has permission to approve certificate request.CertificateSigningPerforms additional authorization checks to ensure the signing user has permission to sign certificate requests.CertificateSubjectRestrictionRejects any certificate request that specifies a 'group' (or 'organization attribute') of system:masters.LimitRangerEnforces the LimitRange API constraints.MutatingAdmissionWebhookAllows the use of custom controllers through webhooks, these controllers may mutate requests that they review.PodSecurityReplacement for Pod Security Policy, restricts security contexts of deployed Pods.ResourceQuotaEnforces resource quotas to prevent over-usage of resources.ValidatingAdmissionWebhookAllows the use of custom controllers through webhooks, these controllers do not mutate requests that it reviews.The second group includes plugins that are not enabled by default but are in general availability state and are recommended to improve your security posture:DenyServiceExternalIPsRejects all net-new usage of the Service.spec.externalIPs field. This is a mitigation for CVE-2020-8554: Man in the middle using LoadBalancer or ExternalIPs.NodeRestrictionRestricts kubelet's permissions to only modify the pods API resources they own or the node API resource that represent themselves. It also prevents kubelet from using the node-restriction.kubernetes.io/ annotation, which can be used by an attacker with access to the kubelet's credentials to influence pod placement to the controlled node.The third group includes plugins that are not enabled by default but could be considered for certain use cases:AlwaysPullImagesEnforces the usage of the latest version of a tagged image and ensures that the deployer has permissions to use the image.ImagePolicyWebhookAllows enforcing additional controls for images through webhooks.What's nextPrivilege escalation via Pod creation warns you about a specific access control risk; check how you're managing that threat.If you use Kubernetes RBAC, read RBAC Good Practices for further information on authorization.Securing a Cluster for information on protecting a cluster from accidental or malicious access.Cluster Multi-tenancy guide for configuration options recommendations and best practices on multi-tenancy.Blog post "A Closer Look at NSA/CISA Kubernetes Hardening Guidance" for complementary resource on hardening Kubernetes clusters.

Security Checklist


Kubernetes ComponentsAn overview of the key components that make up a Kubernetes cluster.This page provides a high-level overview of the essential components that make up a Kubernetes cluster.The components of a Kubernetes clusterCore ComponentsA Kubernetes cluster consists of a control plane and one or more worker nodes. Here's a brief overview of the main components:Control Plane ComponentsManage the overall state of the cluster:kube-apiserverThe core component server that exposes the Kubernetes HTTP API.etcdConsistent and highly-available key value store for all API server data.kube-schedulerLooks for Pods not yet bound to a node, and assigns each Pod to a suitable node.kube-controller-managerRuns controllers to implement Kubernetes API behavior.cloud-controller-manager (optional)Integrates with underlying cloud provider(s).Node ComponentsRun on every node, maintaining running pods and providing the Kubernetes runtime environment:kubeletEnsures that Pods are running, including their containers.kube-proxy (optional)Maintains network rules on nodes to implement Services.Container runtimeSoftware responsible for running containers. Read Container Runtimes to learn more.🛇 This item links to a third party project or product that is not part of Kubernetes itself. More informationYour cluster may require additional software on each node; for example, you might also run systemd on a Linux node to supervise local components.AddonsAddons extend the functionality of Kubernetes. A few important examples include:DNSFor cluster-wide DNS resolution.Web UI (Dashboard)For cluster management via a web interface.Container Resource MonitoringFor collecting and storing container metrics.Cluster-level LoggingFor saving container logs to a central log store.Flexibility in ArchitectureKubernetes allows for flexibility in how these components are deployed and managed. The architecture can be adapted to various needs, from small development environments to large-scale production deployments.For more detailed information about each component and various ways to configure your cluster architecture, see the Cluster Architecture page.Items on this page refer to third party products or projects that provide functionality required by Kubernetes. The Kubernetes project authors aren't responsible for those third-party products or projects. See the CNCF website guidelines for more details.You should read the content guide before proposing a change that adds an extra third-party link.

Kubernetes Components


kubectl certificate approveSynopsisApprove a certificate signing request.kubectl certificate approve allows a cluster admin to approve a certificate signing request (CSR). This action tells a certificate signing controller to issue a certificate to the requester with the attributes requested in the CSR.SECURITY NOTICE: Depending on the requested attributes, the issued certificate can potentially grant a requester access to cluster resources or to authenticate as a requested identity. Before approving a CSR, ensure you understand what the signed certificate can do.kubectl certificate approve (-f FILENAME | NAME) Examples # Approve CSR 'csr-sqgzp' kubectl certificate approve csr-sqgzp Options--allow-missing-template-keys Default: trueIf true, ignore any errors in templates when a field or map key is missing in the template. Only applies to golang and jsonpath output formats.-f, --filename stringsFilename, directory, or URL to files identifying the resource to update--forceUpdate the CSR even if it is already approved.-h, --helphelp for approve-k, --kustomize stringProcess the kustomization directory. This flag can't be used together with -f or -R.-o, --output stringOutput format. One of: (json, yaml, name, go-template, go-template-file, template, templatefile, jsonpath, jsonpath-as-json, jsonpath-file).-R, --recursiveProcess the directory used in -f, --filename recursively. Useful when you want to manage related manifests organized within the same directory.--show-managed-fieldsIf true, keep the managedFields when printing objects in JSON or YAML format.--template stringTemplate string or path to template file to use when -o=go-template, -o=go-template-file. The template format is golang templates [http://golang.org/pkg/text/template/#pkg-overview].Parent Options Inherited--as stringUsername to impersonate for the operation. User could be a regular user or a service account in a namespace.--as-group stringsGroup to impersonate for the operation, this flag can be repeated to specify multiple groups.--as-uid stringUID to impersonate for the operation.--cache-dir string Default: "$HOME/.kube/cache"Default cache directory--certificate-authority stringPath to a cert file for the certificate authority--client-certificate stringPath to a client certificate file for TLS--client-key stringPath to a client key file for TLS--cluster stringThe name of the kubeconfig cluster to use--context stringThe name of the kubeconfig context to use--disable-compressionIf true, opt-out of response compression for all requests to the server--insecure-skip-tls-verifyIf true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure--kubeconfig stringPath to the kubeconfig file to use for CLI requests.--kuberc stringPath to the kuberc file to use for preferences. This can be disabled by exporting KUBECTL_KUBERC=false feature gate or turning off the feature KUBERC=off.--match-server-versionRequire server version to match client version-n, --namespace stringIf present, the namespace scope for this CLI request--password stringPassword for basic authentication to the API server--profile string Default: "none"Name of profile to capture. One of (none|cpu|heap|goroutine|threadcreate|block|mutex)--profile-output string Default: "profile.pprof"Name of the file to write the profile to--request-timeout string Default: "0"The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests.-s, --server stringThe address and port of the Kubernetes API server--storage-driver-buffer-duration duration Default: 1m0sWrites in the storage driver will be buffered for this duration, and committed to the non memory backends as a single transaction--storage-driver-db string Default: "cadvisor"database name--storage-driver-host string Default: "localhost:8086"database host:port--storage-driver-password string Default: "root"database password--storage-driver-secureuse secure connection with database--storage-driver-table string Default: "stats"table name--storage-driver-user string Default: "root"database username--tls-server-name stringServer name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used--token stringBearer token for authentication to the API server--user stringThe name of the kubeconfig user to use--username stringUsername for basic authentication to the API server--version version[=true]--version, --version=raw prints version information and quits; --version=vX.Y.Z... sets the reported version--warnings-as-errorsTreat warnings received from the server as errors and exit with a non-zero exit codeSee Alsokubectl certificate - Modify certificate resourcesThis page is automatically generated.If you plan to report an issue with this page, mention that the page is auto-generated in your issue description. The fix may need to happen elsewhere in the Kubernetes project.

kubectl certificate approve


Custom ResourcesCustom resources are extensions of the Kubernetes API. This page discusses when to add a custom resource to your Kubernetes cluster and when to use a standalone service. It describes the two methods for adding custom resources and how to choose between them.Custom resourcesA resource is an endpoint in the Kubernetes API that stores a collection of API objects of a certain kind; for example, the built-in pods resource contains a collection of Pod objects.A custom resource is an extension of the Kubernetes API that is not necessarily available in a default Kubernetes installation. It represents a customization of a particular Kubernetes installation. However, many core Kubernetes functions are now built using custom resources, making Kubernetes more modular.Custom resources can appear and disappear in a running cluster through dynamic registration, and cluster admins can update custom resources independently of the cluster itself. Once a custom resource is installed, users can create and access its objects using kubectl, just as they do for built-in resources like Pods.Custom controllersOn their own, custom resources let you store and retrieve structured data. When you combine a custom resource with a custom controller, custom resources provide a true declarative API.The Kubernetes declarative API enforces a separation of responsibilities. You declare the desired state of your resource. The Kubernetes controller keeps the current state of Kubernetes objects in sync with your declared desired state. This is in contrast to an imperative API, where you instruct a server what to do.You can deploy and update a custom controller on a running cluster, independently of the cluster's lifecycle. Custom controllers can work with any kind of resource, but they are especially effective when combined with custom resources. The Operator pattern combines custom resources and custom controllers. You can use custom controllers to encode domain knowledge for specific applications into an extension of the Kubernetes API.Should I add a custom resource to my Kubernetes cluster?When creating a new API, consider whether to aggregate your API with the Kubernetes cluster APIs or let your API stand alone.Consider API aggregation if:Prefer a stand-alone API if:Your API is Declarative.Your API does not fit the Declarative model.You want your new types to be readable and writable using kubectl.kubectl support is not requiredYou want to view your new types in a Kubernetes UI, such as dashboard, alongside built-in types.Kubernetes UI support is not required.You are developing a new API.You already have a program that serves your API and works well.You are willing to accept the format restriction that Kubernetes puts on REST resource paths, such as API Groups and Namespaces. (See the API Overview.)You need to have specific REST paths to be compatible with an already defined REST API.Your resources are naturally scoped to a cluster or namespaces of a cluster.Cluster or namespace scoped resources are a poor fit; you need control over the specifics of resource paths.You want to reuse Kubernetes API support features.You don't need those features.Declarative APIsIn a Declarative API, typically:Your API consists of a relatively small number of relatively small objects (resources).The objects define configuration of applications or infrastructure.The objects are updated relatively infrequently.Humans often need to read and write the objects.The main operations on the objects are CRUD-y (creating, reading, updating and deleting).Transactions across objects are not required: the API represents a desired state, not an exact state.Imperative APIs are not declarative. Signs that your API might not be declarative include:The client says "do this", and then gets a synchronous response back when it is done.The client says "do this", and then gets an operation ID back, and has to check a separate Operation object to determine completion of the request.You talk about Remote Procedure Calls (RPCs).Directly storing large amounts of data; for example, > a few kB per object, or > 1000s of objects.High bandwidth access (10s of requests per second sustained) needed.Store end-user data (such as images, PII, etc.) or other large-scale data processed by applications.The natural operations on the objects are not CRUD-y.The API is not easily modeled as objects.You chose to represent pending operations with an operation ID or an operation object.Should I use a ConfigMap or a custom resource?Use a ConfigMap if any of the following apply:There is an existing, well-documented configuration file format, such as a mysql.cnf or pom.xml.You want to put the entire configuration into one key of a ConfigMap.The main use of the configuration file is for a program running in a Pod on your cluster to consume the file to configure itself.Consumers of the file prefer to consume via file in a Pod or environment variable in a pod, rather than the Kubernetes API.You want to perform rolling updates via Deployment, etc., when the file is updated.Note:Use a Secret for sensitive data, which is similar to a ConfigMap but more secure.Use a custom resource (CRD or Aggregated API) if most of the following apply:You want to use Kubernetes client libraries and CLIs to create and update the new resource.You want top-level support from kubectl; for example, kubectl get my-object object-name.You want to build new automation that watches for updates on the new object, and then CRUD other objects, or vice versa.You want to write automation that handles updates to the object.You want to use Kubernetes API conventions like .spec, .status, and .metadata.You want the object to be an abstraction over a collection of controlled resources, or a summarization of other resources.Adding custom resourcesKubernetes provides two ways to add custom resources to your cluster:CRDs are simple and can be created without any programming.API Aggregation requires programming, but allows more control over API behaviors like how data is stored and conversion between API versions.Kubernetes provides these two options to meet the needs of different users, so that neither ease of use nor flexibility is compromised.Aggregated APIs are subordinate API servers that sit behind the primary API server, which acts as a proxy. This arrangement is called API Aggregation(AA). To users, the Kubernetes API appears extended.CRDs allow users to create new types of resources without adding another API server. You do not need to understand API Aggregation to use CRDs.Regardless of how they are installed, the new resources are referred to as Custom Resources to distinguish them from built-in Kubernetes resources (like pods).Note:Avoid using a Custom Resource as data storage for application, end user, or monitoring data: architecture designs that store application data within the Kubernetes API typically represent a design that is too closely coupled.Architecturally, cloud native application architectures favor loose coupling between components. If part of your workload requires a backing service for its routine operation, run that backing service as a component or consume it as an external service. This way, your workload does not rely on the Kubernetes API for its normal operation.CustomResourceDefinitionsThe CustomResourceDefinition API resource allows you to define custom resources. Defining a CRD object creates a new custom resource with a name and schema that you specify. The Kubernetes API serves and handles the storage of your custom resource. The name of the CRD object itself must be a valid DNS subdomain name derived from the defined resource name and its API group; see how to create a CRD for more details. Further, the name of an object whose kind/resource is defined by a CRD must also be a valid DNS subdomain name.This frees you from writing your own API server to handle the custom resource, but the generic nature of the implementation means you have less flexibility than with API server aggregation.Refer to the custom controller example for an example of how to register a new custom resource, work with instances of your new resource type, and use a controller to handle events.API server aggregationUsually, each resource in the Kubernetes API requires code that handles REST requests and manages persistent storage of objects. The main Kubernetes API server handles built-in resources like pods and services, and can also generically handle custom resources through CRDs.The aggregation layer allows you to provide specialized implementations for your custom resources by writing and deploying your own API server. The main API server delegates requests to your API server for the custom resources that you handle, making them available to all of its clients.Choosing a method for adding custom resourcesCRDs are easier to use. Aggregated APIs are more flexible. Choose the method that best meets your needs.Typically, CRDs are a good fit if:You have a handful of fieldsYou are using the resource within your company, or as part of a small open-source project (as opposed to a commercial product)Comparing ease of useCRDs are easier to create than Aggregated APIs.CRDsAggregated APIDo not require programming. Users can choose any language for a CRD controller.Requires programming and building binary and image.No additional service to run; CRDs are handled by API server.An additional service to create and that could fail.No ongoing support once the CRD is created. Any bug fixes are picked up as part of normal Kubernetes Master upgrades.May need to periodically pickup bug fixes from upstream and rebuild and update the Aggregated API server.No need to handle multiple versions of your API; for example, when you control the client for this resource, you can upgrade it in sync with the API.You need to handle multiple versions of your API; for example, when developing an extension to share with the world.Advanced features and flexibilityAggregated APIs offer more advanced API features and customization of other features; for example, the storage layer.FeatureDescriptionCRDsAggregated APIValidationHelp users prevent errors and allow you to evolve your API independently of your clients. These features are most useful when there are many clients who can't all update at the same time.Yes. Most validation can be specified in the CRD using OpenAPI v3.0 validation. CRDValidationRatcheting feature gate allows failing validations specified using OpenAPI also can be ignored if the failing part of the resource was unchanged. Any other validations supported by addition of a Validating Webhook.Yes, arbitrary validation checksDefaultingSee aboveYes, either via OpenAPI v3.0 validation default keyword (GA in 1.17), or via a Mutating Webhook (though this will not be run when reading from etcd for old objects).YesMulti-versioningAllows serving the same object through two API versions. Can help ease API changes like renaming fields. Less important if you control your client versions.YesYesCustom StorageIf you need storage with a different performance mode (for example, a time-series database instead of key-value store) or isolation for security (for example, encryption of sensitive information, etc.)NoYesCustom Business LogicPerform arbitrary checks or actions when creating, reading, updating or deleting an objectYes, using Webhooks.YesScale SubresourceAllows systems like HorizontalPodAutoscaler and PodDisruptionBudget interact with your new resourceYesYesStatus SubresourceAllows fine-grained access control where user writes the spec section and the controller writes the status section. Allows incrementing object Generation on custom resource data mutation (requires separate spec and status sections in the resource)YesYesOther SubresourcesAdd operations other than CRUD, such as "logs" or "exec".NoYesstrategic-merge-patchThe new endpoints support PATCH with Content-Type: application/strategic-merge-patch+json. Useful for updating objects that may be modified both locally, and by the server. For more information, see "Update API Objects in Place Using kubectl patch"NoYesProtocol BuffersThe new resource supports clients that want to use Protocol BuffersNoYesOpenAPI SchemaIs there an OpenAPI (swagger) schema for the types that can be dynamically fetched from the server? Is the user protected from misspelling field names by ensuring only allowed fields are set? Are types enforced (in other words, don't put an int in a string field?)Yes, based on the OpenAPI v3.0 validation schema (GA in 1.16).YesInstance NameDoes this extension mechanism impose any constraints on the names of objects whose kind/resource is defined this way?Yes, such an object's name must be a valid DNS subdomain name.NoCommon FeaturesWhen you create a custom resource, either via a CRD or an AA, you get many features for your API, compared to implementing it outside the Kubernetes platform:FeatureWhat it doesCRUDThe new endpoints support CRUD basic operations via HTTP and kubectlWatchThe new endpoints support Kubernetes Watch operations via HTTPDiscoveryClients like kubectl and dashboard automatically offer list, display, and field edit operations on your resourcesjson-patchThe new endpoints support PATCH with Content-Type: application/json-patch+jsonmerge-patchThe new endpoints support PATCH with Content-Type: application/merge-patch+jsonHTTPSThe new endpoints uses HTTPSBuilt-in AuthenticationAccess to the extension uses the core API server (aggregation layer) for authenticationBuilt-in AuthorizationAccess to the extension can reuse the authorization used by the core API server; for example, RBAC.FinalizersBlock deletion of extension resources until external cleanup happens.Admission WebhooksSet default values and validate extension resources during any create/update/delete operation.UI/CLI DisplayKubectl, dashboard can display extension resources.Unset versus EmptyClients can distinguish unset fields from zero-valued fields.Client Libraries GenerationKubernetes provides generic client libraries, as well as tools to generate type-specific client libraries.Labels and annotationsCommon metadata across objects that tools know how to edit for core and custom resources.Preparing to install a custom resourceThere are several points to be aware of before adding a custom resource to your cluster.Third party code and new points of failureWhile creating a CRD does not automatically add any new points of failure (for example, by causing third party code to run on your API server), packages (for example, Charts) or other installation bundles often include CRDs as well as a Deployment of third-party code that implements the business logic for a new custom resource.Installing an Aggregated API server always involves running a new Deployment.StorageCustom resources consume storage space in the same way that ConfigMaps do. Creating too many custom resources may overload your API server's storage space.Aggregated API servers may use the same storage as the main API server, in which case the same warning applies.Authentication, authorization, and auditingCRDs always use the same authentication, authorization, and audit logging as the built-in resources of your API server.If you use RBAC for authorization, most RBAC roles will not grant access to the new resources (except the cluster-admin role or any role created with wildcard rules). You'll need to explicitly grant access to the new resources. CRDs and Aggregated APIs often come bundled with new role definitions for the types they add.Aggregated API servers may or may not use the same authentication, authorization, and auditing as the primary API server.Accessing a custom resourceKubernetes client libraries can be used to access custom resources. Not all client libraries support custom resources. The Go and Python client libraries do.When you add a custom resource, you can access it using:kubectlThe Kubernetes dynamic client.A REST client that you write.A client generated using Kubernetes client generation tools (generating one is an advanced undertaking, but some projects may provide a client along with the CRD or AA).Custom resource field selectorsField Selectors let clients select custom resources based on the value of one or more resource fields.All custom resources support the metadata.name and metadata.namespace field selectors.Fields declared in a CustomResourceDefinition may also be used with field selectors when included in the spec.versions[*].selectableFields field of the CustomResourceDefinition.Selectable fields for custom resourcesFEATURE STATE: Kubernetes v1.32 [stable] (enabled by default: true)The spec.versions[*].selectableFields field of a CustomResourceDefinition may be used to declare which other fields in a custom resource may be used in field selectors.The following example adds the .spec.color and .spec.size fields as selectable fields.customresourcedefinition/shirt-resource-definition.yaml apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: name: shirts.stable.example.com spec: group: stable.example.com scope: Namespaced names: plural: shirts singular: shirt kind: Shirt versions: - name: v1 served: true storage: true schema: openAPIV3Schema: type: object properties: spec: type: object properties: color: type: string size: type: string selectableFields: - jsonPath: .spec.color - jsonPath: .spec.size additionalPrinterColumns: - jsonPath: .spec.color name: Color type: string - jsonPath: .spec.size name: Size type: string Field selectors can then be used to get only resources with a color of blue:kubectl get shirts.stable.example.com --field-selector spec.color=blue The output should be:NAME COLOR SIZE example1 blue S example2 blue M What's nextLearn how to Extend the Kubernetes API with the aggregation layer.Learn how to Extend the Kubernetes API with CustomResourceDefinition.

Custom Resources


Delete a StatefulSetThis task shows you how to delete a StatefulSet.Before you beginThis task assumes you have an application running on your cluster represented by a StatefulSet.Deleting a StatefulSetYou can delete a StatefulSet in the same way you delete other resources in Kubernetes: use the kubectl delete command, and specify the StatefulSet either by file or by name.kubectl delete -f <file.yaml> kubectl delete statefulsets <statefulset-name> You may need to delete the associated headless service separately after the StatefulSet itself is deleted.kubectl delete service <service-name> When deleting a StatefulSet through kubectl, the StatefulSet scales down to 0. All Pods that are part of this workload are also deleted. If you want to delete only the StatefulSet and not the Pods, use --cascade=orphan. For example:kubectl delete -f <file.yaml> --cascade=orphan By passing --cascade=orphan to kubectl delete, the Pods managed by the StatefulSet are left behind even after the StatefulSet object itself is deleted. If the pods have a label app.kubernetes.io/name=MyApp, you can then delete them as follows:kubectl delete pods -l app.kubernetes.io/name=MyApp Persistent VolumesDeleting the Pods in a StatefulSet will not delete the associated volumes. This is to ensure that you have the chance to copy data off the volume before deleting it. Deleting the PVC after the pods have terminated might trigger deletion of the backing Persistent Volumes depending on the storage class and reclaim policy. You should never assume ability to access a volume after claim deletion.Note:Use caution when deleting a PVC, as it may lead to data loss.Complete deletion of a StatefulSetTo delete everything in a StatefulSet, including the associated pods, you can run a series of commands similar to the following:grace=$(kubectl get pods <stateful-set-pod> --template '{{.spec.terminationGracePeriodSeconds}}') kubectl delete statefulset -l app.kubernetes.io/name=MyApp sleep $grace kubectl delete pvc -l app.kubernetes.io/name=MyApp In the example above, the Pods have the label app.kubernetes.io/name=MyApp; substitute your own label as appropriate.Force deletion of StatefulSet podsIf you find that some pods in your StatefulSet are stuck in the 'Terminating' or 'Unknown' states for an extended period of time, you may need to manually intervene to forcefully delete the pods from the apiserver. This is a potentially dangerous task. Refer to Force Delete StatefulSet Pods for details.What's nextLearn more about force deleting StatefulSet Pods.

Delete a StatefulSet


Install ToolsSet up Kubernetes tools on your computer.kubectlThe Kubernetes command-line tool, kubectl, allows you to run commands against Kubernetes clusters. You can use kubectl to deploy applications, inspect and manage cluster resources, and view logs. For more information including a complete list of kubectl operations, see the kubectl reference documentation.kubectl is installable on a variety of Linux platforms, macOS and Windows. Find your preferred operating system below.Install kubectl on LinuxInstall kubectl on macOSInstall kubectl on Windowskindkind lets you run Kubernetes on your local computer. This tool requires that you have either Docker or Podman installed.The kind Quick Start page shows you what you need to do to get up and running with kind.View kind Quick Start GuideminikubeLike kind, minikube is a tool that lets you run Kubernetes locally. minikube runs an all-in-one or a multi-node local Kubernetes cluster on your personal computer (including Windows, macOS and Linux PCs) so that you can try out Kubernetes, or for daily development work.You can follow the official Get Started! guide if your focus is on getting the tool installed.View minikube Get Started! GuideOnce you have minikube working, you can use it to run a sample application.kubeadmYou can use the kubeadm tool to create and manage Kubernetes clusters. It performs the actions necessary to get a minimum viable, secure cluster up and running in a user friendly way.Installing kubeadm shows you how to install kubeadm. Once installed, you can use it to create a cluster.View kubeadm Install Guide

Install Tools


kubectl scaleSynopsisSet a new size for a deployment, replica set, replication controller, or stateful set.Scale also allows users to specify one or more preconditions for the scale action.If --current-replicas or --resource-version is specified, it is validated before the scale is attempted, and it is guaranteed that the precondition holds true when the scale is sent to the server.kubectl scale [--resource-version=version] [--current-replicas=count] --replicas=COUNT (-f FILENAME | TYPE NAME) Examples # Scale a replica set named 'foo' to 3 kubectl scale --replicas=3 rs/foo # Scale a resource identified by type and name specified in "foo.yaml" to 3 kubectl scale --replicas=3 -f foo.yaml # If the deployment named mysql's current size is 2, scale mysql to 3 kubectl scale --current-replicas=2 --replicas=3 deployment/mysql # Scale multiple replication controllers kubectl scale --replicas=5 rc/example1 rc/example2 rc/example3 # Scale stateful set named 'web' to 3 kubectl scale --replicas=3 statefulset/web Options--allSelect all resources in the namespace of the specified resource types--allow-missing-template-keys Default: trueIf true, ignore any errors in templates when a field or map key is missing in the template. Only applies to golang and jsonpath output formats.--current-replicas int Default: -1Precondition for current size. Requires that the current size of the resource match this value in order to scale. -1 (default) for no condition.--dry-run string[="unchanged"] Default: "none"Must be "none", "server", or "client". If client strategy, only print the object that would be sent, without sending it. If server strategy, submit server-side request without persisting the resource.-f, --filename stringsFilename, directory, or URL to files identifying the resource to set a new size-h, --helphelp for scale-k, --kustomize stringProcess the kustomization directory. This flag can't be used together with -f or -R.-o, --output stringOutput format. One of: (json, yaml, name, go-template, go-template-file, template, templatefile, jsonpath, jsonpath-as-json, jsonpath-file).-R, --recursiveProcess the directory used in -f, --filename recursively. Useful when you want to manage related manifests organized within the same directory.--replicas intThe new desired number of replicas. Required.--resource-version stringPrecondition for resource version. Requires that the current resource version match this value in order to scale.-l, --selector stringSelector (label query) to filter on, supports '=', '==', '!=', 'in', 'notin'.(e.g. -l key1=value1,key2=value2,key3 in (value3)). Matching objects must satisfy all of the specified label constraints.--show-managed-fieldsIf true, keep the managedFields when printing objects in JSON or YAML format.--template stringTemplate string or path to template file to use when -o=go-template, -o=go-template-file. The template format is golang templates [http://golang.org/pkg/text/template/#pkg-overview].--timeout durationThe length of time to wait before giving up on a scale operation, zero means don't wait. Any other values should contain a corresponding time unit (e.g. 1s, 2m, 3h).Parent Options Inherited--as stringUsername to impersonate for the operation. User could be a regular user or a service account in a namespace.--as-group stringsGroup to impersonate for the operation, this flag can be repeated to specify multiple groups.--as-uid stringUID to impersonate for the operation.--cache-dir string Default: "$HOME/.kube/cache"Default cache directory--certificate-authority stringPath to a cert file for the certificate authority--client-certificate stringPath to a client certificate file for TLS--client-key stringPath to a client key file for TLS--cluster stringThe name of the kubeconfig cluster to use--context stringThe name of the kubeconfig context to use--disable-compressionIf true, opt-out of response compression for all requests to the server--insecure-skip-tls-verifyIf true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure--kubeconfig stringPath to the kubeconfig file to use for CLI requests.--kuberc stringPath to the kuberc file to use for preferences. This can be disabled by exporting KUBECTL_KUBERC=false feature gate or turning off the feature KUBERC=off.--match-server-versionRequire server version to match client version-n, --namespace stringIf present, the namespace scope for this CLI request--password stringPassword for basic authentication to the API server--profile string Default: "none"Name of profile to capture. One of (none|cpu|heap|goroutine|threadcreate|block|mutex)--profile-output string Default: "profile.pprof"Name of the file to write the profile to--request-timeout string Default: "0"The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests.-s, --server stringThe address and port of the Kubernetes API server--storage-driver-buffer-duration duration Default: 1m0sWrites in the storage driver will be buffered for this duration, and committed to the non memory backends as a single transaction--storage-driver-db string Default: "cadvisor"database name--storage-driver-host string Default: "localhost:8086"database host:port--storage-driver-password string Default: "root"database password--storage-driver-secureuse secure connection with database--storage-driver-table string Default: "stats"table name--storage-driver-user string Default: "root"database username--tls-server-name stringServer name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used--token stringBearer token for authentication to the API server--user stringThe name of the kubeconfig user to use--username stringUsername for basic authentication to the API server--version version[=true]--version, --version=raw prints version information and quits; --version=vX.Y.Z... sets the reported version--warnings-as-errorsTreat warnings received from the server as errors and exit with a non-zero exit codeSee Alsokubectl - kubectl controls the Kubernetes cluster managerThis page is automatically generated.If you plan to report an issue with this page, mention that the page is auto-generated in your issue description. The fix may need to happen elsewhere in the Kubernetes project.

kubectl scale


Upgrading Linux nodesThis page explains how to upgrade a Linux Worker Nodes created with kubeadm.Before you beginYou need to have shell access to all the nodes, and the kubectl command-line tool must be configured to communicate with your cluster. It is recommended to run this tutorial on a cluster with at least two nodes that are not acting as control plane hosts.To check the version, enter kubectl version.Familiarize yourself with the process for upgrading the rest of your kubeadm cluster. You will want to upgrade the control plane nodes before upgrading your Linux Worker nodes.Changing the package repositoryIf you're using the community-owned package repositories (pkgs.k8s.io), you need to enable the package repository for the desired Kubernetes minor release. This is explained in Changing the Kubernetes package repository document.Note: The legacy package repositories (apt.kubernetes.io and yum.kubernetes.io) have been deprecated and frozen starting from September 13, 2023. Using the new package repositories hosted at pkgs.k8s.io is strongly recommended and required in order to install Kubernetes versions released after September 13, 2023. The deprecated legacy repositories, and their contents, might be removed at any time in the future and without a further notice period. The new package repositories provide downloads for Kubernetes versions starting with v1.24.0.Upgrading worker nodesUpgrade kubeadmUpgrade kubeadm:# replace x in 1.34.x-* with the latest patch version sudo apt-mark unhold kubeadm && \ sudo apt-get update && sudo apt-get install -y kubeadm='1.34.x-*' && \ sudo apt-mark hold kubeadm For systems with DNF:# replace x in 1.34.x-* with the latest patch version sudo yum install -y kubeadm-'1.34.x-*' --disableexcludes=kubernetes For systems with DNF5:# replace x in 1.34.x-* with the latest patch version sudo yum install -y kubeadm-'1.34.x-*' --setopt=disable_excludes=kubernetes Call "kubeadm upgrade"For worker nodes this upgrades the local kubelet configuration:sudo kubeadm upgrade node Drain the nodePrepare the node for maintenance by marking it unschedulable and evicting the workloads:# execute this command on a control plane node # replace <node-to-drain> with the name of your node you are draining kubectl drain <node-to-drain> --ignore-daemonsets Upgrade kubelet and kubectlUpgrade the kubelet and kubectl:# replace x in 1.34.x-* with the latest patch version sudo apt-mark unhold kubelet kubectl && \ sudo apt-get update && sudo apt-get install -y kubelet='1.34.x-*' kubectl='1.34.x-*' && \ sudo apt-mark hold kubelet kubectl For systems with DNF:# replace x in 1.34.x-* with the latest patch version sudo yum install -y kubelet-'1.34.x-*' kubectl-'1.34.x-*' --disableexcludes=kubernetes For systems with DNF5:# replace x in 1.34.x-* with the latest patch version sudo yum install -y kubelet-'1.34.x-*' kubectl-'1.34.x-*' --setopt=disable_excludes=kubernetes Restart the kubelet:sudo systemctl daemon-reload sudo systemctl restart kubelet Uncordon the nodeBring the node back online by marking it schedulable:# execute this command on a control plane node # replace <node-to-uncordon> with the name of your node kubectl uncordon <node-to-uncordon> What's nextSee how to Upgrade Windows nodes.

Upgrading Linux nodes


kubectl top nodeSynopsisDisplay resource (CPU/memory) usage of nodes.The top-node command allows you to see the resource consumption of nodes.kubectl top node [NAME | -l label] Examples # Show metrics for all nodes kubectl top node # Show metrics for a given node kubectl top node NODE_NAME Options-h, --helphelp for node--no-headersIf present, print output without headers-l, --selector stringSelector (label query) to filter on, supports '=', '==', '!=', 'in', 'notin'.(e.g. -l key1=value1,key2=value2,key3 in (value3)). Matching objects must satisfy all of the specified label constraints.--show-capacityPrint node resources based on Capacity instead of Allocatable(default) of the nodes.--show-swapPrint node resources related to swap memory.--sort-by stringIf non-empty, sort nodes list using specified field. The field can be either 'cpu' or 'memory'.--use-protocol-buffers Default: trueEnables using protocol-buffers to access Metrics API.Parent Options Inherited--as stringUsername to impersonate for the operation. User could be a regular user or a service account in a namespace.--as-group stringsGroup to impersonate for the operation, this flag can be repeated to specify multiple groups.--as-uid stringUID to impersonate for the operation.--cache-dir string Default: "$HOME/.kube/cache"Default cache directory--certificate-authority stringPath to a cert file for the certificate authority--client-certificate stringPath to a client certificate file for TLS--client-key stringPath to a client key file for TLS--cluster stringThe name of the kubeconfig cluster to use--context stringThe name of the kubeconfig context to use--disable-compressionIf true, opt-out of response compression for all requests to the server--insecure-skip-tls-verifyIf true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure--kubeconfig stringPath to the kubeconfig file to use for CLI requests.--kuberc stringPath to the kuberc file to use for preferences. This can be disabled by exporting KUBECTL_KUBERC=false feature gate or turning off the feature KUBERC=off.--match-server-versionRequire server version to match client version-n, --namespace stringIf present, the namespace scope for this CLI request--password stringPassword for basic authentication to the API server--profile string Default: "none"Name of profile to capture. One of (none|cpu|heap|goroutine|threadcreate|block|mutex)--profile-output string Default: "profile.pprof"Name of the file to write the profile to--request-timeout string Default: "0"The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests.-s, --server stringThe address and port of the Kubernetes API server--storage-driver-buffer-duration duration Default: 1m0sWrites in the storage driver will be buffered for this duration, and committed to the non memory backends as a single transaction--storage-driver-db string Default: "cadvisor"database name--storage-driver-host string Default: "localhost:8086"database host:port--storage-driver-password string Default: "root"database password--storage-driver-secureuse secure connection with database--storage-driver-table string Default: "stats"table name--storage-driver-user string Default: "root"database username--tls-server-name stringServer name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used--token stringBearer token for authentication to the API server--user stringThe name of the kubeconfig user to use--username stringUsername for basic authentication to the API server--version version[=true]--version, --version=raw prints version information and quits; --version=vX.Y.Z... sets the reported version--warnings-as-errorsTreat warnings received from the server as errors and exit with a non-zero exit codeSee Alsokubectl top - Display resource (CPU/memory) usageThis page is automatically generated.If you plan to report an issue with this page, mention that the page is auto-generated in your issue description. The fix may need to happen elsewhere in the Kubernetes project.

kubectl top node


kubectl autoscaleSynopsisCreates an autoscaler that automatically chooses and sets the number of pods that run in a Kubernetes cluster. The command will attempt to use the autoscaling/v2 API first, in case of an error, it will fall back to autoscaling/v1 API.Looks up a deployment, replica set, stateful set, or replication controller by name and creates an autoscaler that uses the given resource as a reference. An autoscaler can automatically increase or decrease number of pods deployed within the system as needed.kubectl autoscale (-f FILENAME | TYPE NAME | TYPE/NAME) [--min=MINPODS] --max=MAXPODS [--cpu=CPU] [--memory=MEMORY] Examples # Auto scale a deployment "foo", with the number of pods between 2 and 10, no target CPU utilization specified so a default autoscaling policy will be used kubectl autoscale deployment foo --min=2 --max=10 # Auto scale a replication controller "foo", with the number of pods between 1 and 5, target CPU utilization at 80% kubectl autoscale rc foo --max=5 --cpu=80% # Auto scale a deployment "bar", with the number of pods between 3 and 6, target average CPU of 500m and memory of 200Mi kubectl autoscale deployment bar --min=3 --max=6 --cpu=500m --memory=200Mi # Auto scale a deployment "bar", with the number of pods between 2 and 8, target CPU utilization 60% and memory utilization 70% kubectl autoscale deployment bar --min=3 --max=6 --cpu=60% --memory=70% Options--allow-missing-template-keys Default: trueIf true, ignore any errors in templates when a field or map key is missing in the template. Only applies to golang and jsonpath output formats.--cpu stringTarget CPU utilization over all the pods. When specified as a percentage (e.g."70%" for 70% of requested CPU) it will target average utilization. When specified as quantity (e.g."500m" for 500 milliCPU) it will target average value. Value without units is treated as a quantity with miliCPU being the unit (e.g."500" is "500m").--dry-run string[="unchanged"] Default: "none"Must be "none", "server", or "client". If client strategy, only print the object that would be sent, without sending it. If server strategy, submit server-side request without persisting the resource.--field-manager string Default: "kubectl-autoscale"Name of the manager used to track field ownership.-f, --filename stringsFilename, directory, or URL to files identifying the resource to autoscale.-h, --helphelp for autoscale-k, --kustomize stringProcess the kustomization directory. This flag can't be used together with -f or -R.--max int32 Default: -1The upper limit for the number of pods that can be set by the autoscaler. Required.--memory stringTarget memory utilization over all the pods. When specified as a percentage (e.g."60%" for 60% of requested memory) it will target average utilization. When specified as quantity (e.g."200Mi" for 200 MiB, "1Gi" for 1 GiB) it will target average value. Value without units is treated as a quantity with mebibytes being the unit (e.g."200" is "200Mi").--min int32 Default: -1The lower limit for the number of pods that can be set by the autoscaler. If it's not specified or negative, the server will apply a default value.--name stringThe name for the newly created object. If not specified, the name of the input resource will be used.-o, --output stringOutput format. One of: (json, yaml, name, go-template, go-template-file, template, templatefile, jsonpath, jsonpath-as-json, jsonpath-file).-R, --recursiveProcess the directory used in -f, --filename recursively. Useful when you want to manage related manifests organized within the same directory.--save-configIf true, the configuration of current object will be saved in its annotation. Otherwise, the annotation will be unchanged. This flag is useful when you want to perform kubectl apply on this object in the future.--show-managed-fieldsIf true, keep the managedFields when printing objects in JSON or YAML format.--template stringTemplate string or path to template file to use when -o=go-template, -o=go-template-file. The template format is golang templates [http://golang.org/pkg/text/template/#pkg-overview].Parent Options Inherited--as stringUsername to impersonate for the operation. User could be a regular user or a service account in a namespace.--as-group stringsGroup to impersonate for the operation, this flag can be repeated to specify multiple groups.--as-uid stringUID to impersonate for the operation.--cache-dir string Default: "$HOME/.kube/cache"Default cache directory--certificate-authority stringPath to a cert file for the certificate authority--client-certificate stringPath to a client certificate file for TLS--client-key stringPath to a client key file for TLS--cluster stringThe name of the kubeconfig cluster to use--context stringThe name of the kubeconfig context to use--disable-compressionIf true, opt-out of response compression for all requests to the server--insecure-skip-tls-verifyIf true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure--kubeconfig stringPath to the kubeconfig file to use for CLI requests.--kuberc stringPath to the kuberc file to use for preferences. This can be disabled by exporting KUBECTL_KUBERC=false feature gate or turning off the feature KUBERC=off.--match-server-versionRequire server version to match client version-n, --namespace stringIf present, the namespace scope for this CLI request--password stringPassword for basic authentication to the API server--profile string Default: "none"Name of profile to capture. One of (none|cpu|heap|goroutine|threadcreate|block|mutex)--profile-output string Default: "profile.pprof"Name of the file to write the profile to--request-timeout string Default: "0"The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests.-s, --server stringThe address and port of the Kubernetes API server--storage-driver-buffer-duration duration Default: 1m0sWrites in the storage driver will be buffered for this duration, and committed to the non memory backends as a single transaction--storage-driver-db string Default: "cadvisor"database name--storage-driver-host string Default: "localhost:8086"database host:port--storage-driver-password string Default: "root"database password--storage-driver-secureuse secure connection with database--storage-driver-table string Default: "stats"table name--storage-driver-user string Default: "root"database username--tls-server-name stringServer name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used--token stringBearer token for authentication to the API server--user stringThe name of the kubeconfig user to use--username stringUsername for basic authentication to the API server--version version[=true]--version, --version=raw prints version information and quits; --version=vX.Y.Z... sets the reported version--warnings-as-errorsTreat warnings received from the server as errors and exit with a non-zero exit codeSee Alsokubectl - kubectl controls the Kubernetes cluster managerThis page is automatically generated.If you plan to report an issue with this page, mention that the page is auto-generated in your issue description. The fix may need to happen elsewhere in the Kubernetes project.

kubectl autoscale


Set Kubelet Parameters Via A Configuration FileBefore you beginSome steps in this page use the jq tool. If you don't have jq, you can install it via your operating system's software sources, or fetch it from https://jqlang.github.io/jq/.Some steps also involve installing curl, which can be installed via your operating system's software sources.A subset of the kubelet's configuration parameters may be set via an on-disk config file, as a substitute for command-line flags.Providing parameters via a config file is the recommended approach because it simplifies node deployment and configuration management.Create the config fileThe subset of the kubelet's configuration that can be configured via a file is defined by the KubeletConfiguration struct.The configuration file must be a JSON or YAML representation of the parameters in this struct. Make sure the kubelet has read permissions on the file.Here is an example of what this file might look like:apiVersion: kubelet.config.k8s.io/v1beta1 kind: KubeletConfiguration address: "192.168.0.8" port: 20250 serializeImagePulls: false evictionHard: memory.available: "100Mi" nodefs.available: "10%" nodefs.inodesFree: "5%" imagefs.available: "15%" imagefs.inodesFree: "5%" In this example, the kubelet is configured with the following settings:address: The kubelet will serve on IP address 192.168.0.8.port: The kubelet will serve on port 20250.serializeImagePulls: Image pulls will be done in parallel.evictionHard: The kubelet will evict Pods under one of the following conditions:When the node's available memory drops below 100MiB.When the node's main filesystem's available space is less than 10%.When the image filesystem's available space is less than 15%.When more than 95% of the node's main filesystem's inodes are in use.Note:In the example, by changing the default value of only one parameter for evictionHard, the default values of other parameters will not be inherited and will be set to zero. In order to provide custom values, you should provide all the threshold values respectively. Alternatively, you can set the MergeDefaultEvictionSettings to true in the kubelet configuration file, if any parameter is changed then the other parameters will inherit their default values instead of 0.The imagefs is an optional filesystem that container runtimes use to store container images and container writable layers.Start a kubelet process configured via the config fileNote:If you use kubeadm to initialize your cluster, use the kubelet-config while creating your cluster with kubeadm init. See configuring kubelet using kubeadm for details.Start the kubelet with the --config flag set to the path of the kubelet's config file. The kubelet will then load its config from this file.Note that command line flags which target the same value as a config file will override that value. This helps ensure backwards compatibility with the command-line API.Note that relative file paths in the kubelet config file are resolved relative to the location of the kubelet config file, whereas relative paths in command line flags are resolved relative to the kubelet's current working directory.Note that some default values differ between command-line flags and the kubelet config file. If --config is provided and the values are not specified via the command line, the defaults for the KubeletConfiguration version apply. In the above example, this version is kubelet.config.k8s.io/v1beta1.Drop-in directory for kubelet configuration filesFEATURE STATE: Kubernetes v1.30 [beta]You can specify a drop-in configuration directory for the kubelet. By default, the kubelet does not look for drop-in configuration files anywhere - you must specify a path. For example: --config-dir=/etc/kubernetes/kubelet.conf.dFor Kubernetes v1.28 to v1.29, you can only specify --config-dir if you also set the environment variable KUBELET_CONFIG_DROPIN_DIR_ALPHA for the kubelet process (the value of that variable does not matter).Note:The suffix of a valid kubelet drop-in configuration file must be .conf. For instance: 99-kubelet-address.confThe kubelet processes files in its config drop-in directory by sorting the entire file name alphanumerically. For instance, 00-kubelet.conf is processed first, and then overridden with a file named 01-kubelet.conf.These files may contain partial configurations but should not be invalid and must include type metadata, specifically apiVersion and kind. Validation is only performed on the final resulting configuration structure stored internally in the kubelet. This offers flexibility in managing and merging kubelet configurations from different sources while preventing undesirable configurations. However, it is important to note that behavior varies based on the data type of the configuration fields.Different data types in the kubelet configuration structure merge differently. See the reference document for more information.Kubelet configuration merging orderOn startup, the kubelet merges configuration from:Feature gates specified over the command line (lowest precedence).The kubelet configuration.Drop-in configuration files, according to sort order.Command line arguments excluding feature gates (highest precedence).Note:The config drop-in dir mechanism for the kubelet is similar but different from how the kubeadm tool allows you to patch configuration. The kubeadm tool uses a specific patching strategy for its configuration, whereas the only patch strategy for kubelet configuration drop-in files is replace. The kubelet determines the order of merges based on sorting the suffixes alphanumerically, and replaces every field present in a higher priority file.Viewing the kubelet configurationSince the configuration could now be spread over multiple files with this feature, if someone wants to inspect the final actuated configuration, they can follow these steps to inspect the kubelet configuration:Start a proxy server using kubectl proxy in your terminal.kubectl proxy Which gives output like:Starting to serve on 127.0.0.1:8001 Open another terminal window and use curl to fetch the kubelet configuration. Replace <node-name> with the actual name of your node:curl -X GET http://127.0.0.1:8001/api/v1/nodes/<node-name>/proxy/configz | jq . { "kubeletconfig": { "enableServer": true, "staticPodPath": "/var/run/kubernetes/static-pods", "syncFrequency": "1m0s", "fileCheckFrequency": "20s", "httpCheckFrequency": "20s", "address": "192.168.1.16", "port": 10250, "readOnlyPort": 10255, "tlsCertFile": "/var/lib/kubelet/pki/kubelet.crt", "tlsPrivateKeyFile": "/var/lib/kubelet/pki/kubelet.key", "rotateCertificates": true, "authentication": { "x509": { "clientCAFile": "/var/run/kubernetes/client-ca.crt" }, "webhook": { "enabled": true, "cacheTTL": "2m0s" }, "anonymous": { "enabled": true } }, "authorization": { "mode": "AlwaysAllow", "webhook": { "cacheAuthorizedTTL": "5m0s", "cacheUnauthorizedTTL": "30s" } }, "registryPullQPS": 5, "registryBurst": 10, "eventRecordQPS": 50, "eventBurst": 100, "enableDebuggingHandlers": true, "healthzPort": 10248, "healthzBindAddress": "127.0.0.1", "oomScoreAdj": -999, "clusterDomain": "cluster.local", "clusterDNS": [ "10.0.0.10" ], "streamingConnectionIdleTimeout": "4h0m0s", "nodeStatusUpdateFrequency": "10s", "nodeStatusReportFrequency": "5m0s", "nodeLeaseDurationSeconds": 40, "imageMinimumGCAge": "2m0s", "imageMaximumGCAge": "0s", "imageGCHighThresholdPercent": 85, "imageGCLowThresholdPercent": 80, "volumeStatsAggPeriod": "1m0s", "cgroupsPerQOS": true, "cgroupDriver": "systemd", "cpuManagerPolicy": "none", "cpuManagerReconcilePeriod": "10s", "memoryManagerPolicy": "None", "topologyManagerPolicy": "none", "topologyManagerScope": "container", "runtimeRequestTimeout": "2m0s", "hairpinMode": "promiscuous-bridge", "maxPods": 110, "podPidsLimit": -1, "resolvConf": "/run/systemd/resolve/resolv.conf", "cpuCFSQuota": true, "cpuCFSQuotaPeriod": "100ms", "nodeStatusMaxImages": 50, "maxOpenFiles": 1000000, "contentType": "application/vnd.kubernetes.protobuf", "kubeAPIQPS": 50, "kubeAPIBurst": 100, "serializeImagePulls": true, "evictionHard": { "imagefs.available": "15%", "memory.available": "100Mi", "nodefs.available": "10%", "nodefs.inodesFree": "5%", "imagefs.inodesFree": "5%" }, "evictionPressureTransitionPeriod": "1m0s", "mergeDefaultEvictionSettings": false, "enableControllerAttachDetach": true, "makeIPTablesUtilChains": true, "iptablesMasqueradeBit": 14, "iptablesDropBit": 15, "featureGates": { "AllAlpha": false }, "failSwapOn": false, "memorySwap": {}, "containerLogMaxSize": "10Mi", "containerLogMaxFiles": 5, "configMapAndSecretChangeDetectionStrategy": "Watch", "enforceNodeAllocatable": [ "pods" ], "volumePluginDir": "/usr/libexec/kubernetes/kubelet-plugins/volume/exec/", "logging": { "format": "text", "flushFrequency": "5s", "verbosity": 3, "options": { "json": { "infoBufferSize": "0" } } }, "enableSystemLogHandler": true, "enableSystemLogQuery": false, "shutdownGracePeriod": "0s", "shutdownGracePeriodCriticalPods": "0s", "enableProfilingHandler": true, "enableDebugFlagsHandler": true, "seccompDefault": false, "memoryThrottlingFactor": 0.9, "registerNode": true, "localStorageCapacityIsolation": true, "containerRuntimeEndpoint": "unix:///var/run/crio/crio.sock" } } What's nextLearn more about kubelet configuration by checking the KubeletConfiguration reference.Learn more about kubelet configuration merging in the reference document.