kube-schedulerSynopsisThe Kubernetes scheduler is a control plane process which assigns Pods to Nodes. The scheduler determines which Nodes are valid placements for each Pod in the scheduling queue according to constraints and available resources. The scheduler then ranks each valid Node and binds the Pod to a suitable Node. Multiple different schedulers may be used within a cluster; kube-scheduler is the reference implementation. See scheduling for more information about scheduling and the kube-scheduler component.kube-scheduler [flags] Options--allow-metric-labels stringToString Default: []The map from metric-label to value allow-list of this label. The key's format is <MetricName>,<LabelName>. The value's format is <allowed_value>,<allowed_value>...e.g. metric1,label1='v1,v2,v3', metric1,label2='v1,v2,v3' metric2,label1='v1,v2,v3'.--allow-metric-labels-manifest stringThe path to the manifest file that contains the allow-list mapping. The format of the file is the same as the flag --allow-metric-labels. Note that the flag --allow-metric-labels will override the manifest file.--authentication-kubeconfig stringkubeconfig file pointing at the 'core' kubernetes server with enough rights to create tokenreviews.authentication.k8s.io. This is optional. If empty, all token requests are considered to be anonymous and no client CA is looked up in the cluster.--authentication-skip-lookupIf false, the authentication-kubeconfig will be used to lookup missing authentication configuration from the cluster.--authentication-token-webhook-cache-ttl duration Default: 10sThe duration to cache responses from the webhook token authenticator.--authentication-tolerate-lookup-failure Default: trueIf true, failures to look up missing authentication configuration from the cluster are not considered fatal. Note that this can result in authentication that treats all requests as anonymous.--authorization-always-allow-paths strings Default: "/healthz,/readyz,/livez"A list of HTTP paths to skip during authorization, i.e. these are authorized without contacting the 'core' kubernetes server.--authorization-kubeconfig stringkubeconfig file pointing at the 'core' kubernetes server with enough rights to create subjectaccessreviews.authorization.k8s.io. This is optional. If empty, all requests not skipped by authorization are forbidden.--authorization-webhook-cache-authorized-ttl duration Default: 10sThe duration to cache 'authorized' responses from the webhook authorizer.--authorization-webhook-cache-unauthorized-ttl duration Default: 10sThe duration to cache 'unauthorized' responses from the webhook authorizer.--bind-address string Default: 0.0.0.0The IP address on which to listen for the --secure-port port. The associated interface(s) must be reachable by the rest of the cluster, and by CLI/web clients. If blank or an unspecified address (0.0.0.0 or ::), all interfaces and IP address families will be used.--cert-dir stringThe directory where the TLS certs are located. If --tls-cert-file and --tls-private-key-file are provided, this flag will be ignored.--client-ca-file stringIf set, any request presenting a client certificate signed by one of the authorities in the client-ca-file is authenticated with an identity corresponding to the CommonName of the client certificate.--config stringThe path to the configuration file.--contention-profiling Default: trueDEPRECATED: enable block profiling, if profiling is enabled. This parameter is ignored if a config file is specified in --config.--disable-http2-servingIf true, HTTP2 serving will be disabled [default=false]--disabled-metrics stringsThis flag provides an escape hatch for misbehaving metrics. You must provide the fully qualified metric name in order to disable it. Disclaimer: disabling metrics is higher in precedence than showing hidden metrics.--emulated-version stringsThe versions different components emulate their capabilities (APIs, features, ...) of.If set, the component will emulate the behavior of this version instead of the underlying binary version.Version format could only be major.minor, for example: '--emulated-version=wardle=1.2,kube=1.31'.Options are: kube=1.31..1.33(default:1.33)If the component is not specified, defaults to "kube"--feature-gates colonSeparatedMultimapStringStringComma-separated list of component:key=value pairs that describe feature gates for alpha/experimental features of different components.If the component is not specified, defaults to "kube". This flag can be repeatedly invoked. For example: --feature-gates 'wardle:featureA=true,wardle:featureB=false' --feature-gates 'kube:featureC=true'Options are:kube:APIResponseCompression=true|false (BETA - default=true)kube:APIServerIdentity=true|false (BETA - default=true)kube:APIServerTracing=true|false (BETA - default=true)kube:APIServingWithRoutine=true|false (ALPHA - default=false)kube:AllAlpha=true|false (ALPHA - default=false)kube:AllBeta=true|false (BETA - default=false)kube:AllowParsingUserUIDFromCertAuth=true|false (BETA - default=true)kube:AllowUnsafeMalformedObjectDeletion=true|false (ALPHA - default=false)kube:AnonymousAuthConfigurableEndpoints=true|false (BETA - default=true)kube:AuthorizeNodeWithSelectors=true|false (BETA - default=true)kube:AuthorizeWithSelectors=true|false (BETA - default=true)kube:CBORServingAndStorage=true|false (ALPHA - default=false)kube:CPUManagerPolicyAlphaOptions=true|false (ALPHA - default=false)kube:CPUManagerPolicyBetaOptions=true|false (BETA - default=true)kube:CSIVolumeHealth=true|false (ALPHA - default=false)kube:ClientsAllowCBOR=true|false (ALPHA - default=false)kube:ClientsPreferCBOR=true|false (ALPHA - default=false)kube:CloudControllerManagerWebhook=true|false (ALPHA - default=false)kube:ClusterTrustBundle=true|false (BETA - default=false)kube:ClusterTrustBundleProjection=true|false (BETA - default=false)kube:ComponentFlagz=true|false (ALPHA - default=false)kube:ComponentStatusz=true|false (ALPHA - default=false)kube:ConcurrentWatchObjectDecode=true|false (BETA - default=false)kube:ConsistentListFromCache=true|false (BETA - default=true)kube:ContainerCheckpoint=true|false (BETA - default=true)kube:ContainerStopSignals=true|false (ALPHA - default=false)kube:ContextualLogging=true|false (BETA - default=true)kube:CoordinatedLeaderElection=true|false (BETA - default=false)kube:CrossNamespaceVolumeDataSource=true|false (ALPHA - default=false)kube:CustomCPUCFSQuotaPeriod=true|false (ALPHA - default=false)kube:DRAAdminAccess=true|false (ALPHA - default=false)kube:DRADeviceTaints=true|false (ALPHA - default=false)kube:DRAPartitionableDevices=true|false (ALPHA - default=false)kube:DRAPrioritizedList=true|false (ALPHA - default=false)kube:DRAResourceClaimDeviceStatus=true|false (BETA - default=true)kube:DeclarativeValidation=true|false (BETA - default=true)kube:DeclarativeValidationTakeover=true|false (BETA - default=false)kube:DeploymentReplicaSetTerminatingReplicas=true|false (ALPHA - default=false)kube:DisableAllocatorDualWrite=true|false (BETA - default=false)kube:DisableCPUQuotaWithExclusiveCPUs=true|false (BETA - default=true)kube:DynamicResourceAllocation=true|false (BETA - default=false)kube:EventedPLEG=true|false (ALPHA - default=false)kube:ExternalServiceAccountTokenSigner=true|false (ALPHA - default=false)kube:GracefulNodeShutdown=true|false (BETA - default=true)kube:GracefulNodeShutdownBasedOnPodPriority=true|false (BETA - default=true)kube:HPAConfigurableTolerance=true|false (ALPHA - default=false)kube:HPAScaleToZero=true|false (ALPHA - default=false)kube:ImageMaximumGCAge=true|false (BETA - default=true)kube:ImageVolume=true|false (BETA - default=false)kube:InOrderInformers=true|false (BETA - default=true)kube:InPlacePodVerticalScaling=true|false (BETA - default=true)kube:InPlacePodVerticalScalingExclusiveCPUs=true|false (ALPHA - default=false)kube:InTreePluginPortworxUnregister=true|false (ALPHA - default=false)kube:InformerResourceVersion=true|false (ALPHA - default=false)kube:JobManagedBy=true|false (BETA - default=true)kube:JobPodReplacementPolicy=true|false (BETA - default=true)kube:KubeletCgroupDriverFromCRI=true|false (BETA - default=true)kube:KubeletCrashLoopBackOffMax=true|false (ALPHA - default=false)kube:KubeletEnsureSecretPulledImages=true|false (ALPHA - default=false)kube:KubeletFineGrainedAuthz=true|false (BETA - default=true)kube:KubeletInUserNamespace=true|false (ALPHA - default=false)kube:KubeletPSI=true|false (ALPHA - default=false)kube:KubeletPodResourcesDynamicResources=true|false (ALPHA - default=false)kube:KubeletPodResourcesGet=true|false (ALPHA - default=false)kube:KubeletSeparateDiskGC=true|false (BETA - default=true)kube:KubeletServiceAccountTokenForCredentialProviders=true|false (ALPHA - default=false)kube:KubeletTracing=true|false (BETA - default=true)kube:ListFromCacheSnapshot=true|false (ALPHA - default=false)kube:LocalStorageCapacityIsolationFSQuotaMonitoring=true|false (BETA - default=false)kube:LoggingAlphaOptions=true|false (ALPHA - default=false)kube:LoggingBetaOptions=true|false (BETA - default=true)kube:MatchLabelKeysInPodTopologySpread=true|false (BETA - default=true)kube:MaxUnavailableStatefulSet=true|false (ALPHA - default=false)kube:MemoryQoS=true|false (ALPHA - default=false)kube:MutableCSINodeAllocatableCount=true|false (ALPHA - default=false)kube:MutatingAdmissionPolicy=true|false (ALPHA - default=false)kube:NodeLogQuery=true|false (BETA - default=false)kube:NodeSwap=true|false (BETA - default=true)kube:OpenAPIEnums=true|false (BETA - default=true)kube:OrderedNamespaceDeletion=true|false (BETA - default=true)kube:PodAndContainerStatsFromCRI=true|false (ALPHA - default=false)kube:PodDeletionCost=true|false (BETA - default=true)kube:PodLevelResources=true|false (ALPHA - default=false)kube:PodLifecycleSleepAction=true|false (BETA - default=true)kube:PodLifecycleSleepActionAllowZero=true|false (BETA - default=true)kube:PodLogsQuerySplitStreams=true|false (ALPHA - default=false)kube:PodObservedGenerationTracking=true|false (ALPHA - default=false)kube:PodReadyToStartContainersCondition=true|false (BETA - default=true)kube:PodTopologyLabelsAdmission=true|false (ALPHA - default=false)kube:PortForwardWebsockets=true|false (BETA - default=true)kube:PreferSameTrafficDistribution=true|false (ALPHA - default=false)kube:ProcMountType=true|false (BETA - default=true)kube:QOSReserved=true|false (ALPHA - default=false)kube:RecoverVolumeExpansionFailure=true|false (BETA - default=true)kube:ReduceDefaultCrashLoopBackOffDecay=true|false (ALPHA - default=false)kube:RelaxedDNSSearchValidation=true|false (BETA - default=true)kube:RelaxedEnvironmentVariableValidation=true|false (BETA - default=true)kube:ReloadKubeletServerCertificateFile=true|false (BETA - default=true)kube:RemoteRequestHeaderUID=true|false (BETA - default=true)kube:ResilientWatchCacheInitialization=true|false (BETA - default=true)kube:ResourceHealthStatus=true|false (ALPHA - default=false)kube:RotateKubeletServerCertificate=true|false (BETA - default=true)kube:RuntimeClassInImageCriApi=true|false (ALPHA - default=false)kube:SELinuxChangePolicy=true|false (BETA - default=true)kube:SELinuxMount=true|false (BETA - default=false)kube:SELinuxMountReadWriteOncePod=true|false (BETA - default=true)kube:SchedulerAsyncPreemption=true|false (BETA - default=true)kube:SchedulerPopFromBackoffQ=true|false (BETA - default=true)kube:SchedulerQueueingHints=true|false (BETA - default=true)kube:SeparateTaintEvictionController=true|false (BETA - default=true)kube:ServiceAccountNodeAudienceRestriction=true|false (BETA - default=true)kube:StorageCapacityScoring=true|false (ALPHA - default=false)kube:StorageVersionAPI=true|false (ALPHA - default=false)kube:StorageVersionHash=true|false (BETA - default=true)kube:StorageVersionMigrator=true|false (ALPHA - default=false)kube:StreamingCollectionEncodingToJSON=true|false (BETA - default=true)kube:StreamingCollectionEncodingToProtobuf=true|false (BETA - default=true)kube:StrictIPCIDRValidation=true|false (ALPHA - default=false)kube:StructuredAuthenticationConfiguration=true|false (BETA - default=true)kube:SupplementalGroupsPolicy=true|false (BETA - default=true)kube:SystemdWatchdog=true|false (BETA - default=true)kube:TopologyManagerPolicyAlphaOptions=true|false (ALPHA - default=false)kube:TopologyManagerPolicyBetaOptions=true|false (BETA - default=true)kube:TranslateStreamCloseWebsocketRequests=true|false (BETA - default=true)kube:UnauthenticatedHTTP2DOSMitigation=true|false (BETA - default=true)kube:UnknownVersionInteroperabilityProxy=true|false (ALPHA - default=false)kube:UserNamespacesPodSecurityStandards=true|false (ALPHA - default=false)kube:UserNamespacesSupport=true|false (BETA - default=true)kube:VolumeAttributesClass=true|false (BETA - default=false)kube:WatchCacheInitializationPostStartHook=true|false (BETA - default=false)kube:WatchList=true|false (BETA - default=false)kube:WatchListClient=true|false (BETA - default=false)kube:WinDSR=true|false (BETA - default=true)kube:WinOverlay=true|false (BETA - default=true)kube:WindowsCPUAndMemoryAffinity=true|false (ALPHA - default=false)kube:WindowsGracefulNodeShutdown=true|false (ALPHA - default=false)-h, --helphelp for kube-scheduler--http2-max-streams-per-connection intThe limit that the server gives to clients for the maximum number of streams in an HTTP/2 connection. Zero means to use golang's default.--kube-api-burst int32 Default: 100DEPRECATED: burst to use while talking with kubernetes apiserver. This parameter is ignored if a config file is specified in --config.--kube-api-content-type string Default: "application/vnd.kubernetes.protobuf"DEPRECATED: content type of requests sent to apiserver. This parameter is ignored if a config file is specified in --config.--kube-api-qps float Default: 50DEPRECATED: QPS to use while talking with kubernetes apiserver. This parameter is ignored if a config file is specified in --config.--kubeconfig stringDEPRECATED: path to kubeconfig file with authorization and master location information. This parameter is ignored if a config file is specified in --config.--leader-elect Default: trueStart a leader election client and gain leadership before executing the main loop. Enable this when running replicated components for high availability.--leader-elect-lease-duration duration Default: 15sThe duration that non-leader candidates will wait after observing a leadership renewal until attempting to acquire leadership of a led but unrenewed leader slot. This is effectively the maximum duration that a leader can be stopped before it is replaced by another candidate. This is only applicable if leader election is enabled.--leader-elect-renew-deadline duration Default: 10sThe interval between attempts by the acting master to renew a leadership slot before it stops leading. This must be less than the lease duration. This is only applicable if leader election is enabled.--leader-elect-resource-lock string Default: "leases"The type of resource object that is used for locking during leader election. Supported options are 'leases'.--leader-elect-resource-name string Default: "kube-scheduler"The name of resource object that is used for locking during leader election.--leader-elect-resource-namespace string Default: "kube-system"The namespace of resource object that is used for locking during leader election.--leader-elect-retry-period duration Default: 2sThe duration the clients should wait between attempting acquisition and renewal of a leadership. This is only applicable if leader election is enabled.--log-flush-frequency duration Default: 5sMaximum number of seconds between log flushes--log-text-info-buffer-size quantity[Alpha] In text format with split output streams, the info messages can be buffered for a while to increase performance. The default value of zero bytes disables buffering. The size can be specified as number of bytes (512), multiples of 1000 (1K), multiples of 1024 (2Ki), or powers of those (3M, 4G, 5Mi, 6Gi). Enable the LoggingAlphaOptions feature gate to use this.--log-text-split-stream[Alpha] In text format, write error messages to stderr and info messages to stdout. The default is to write a single stream to stdout. Enable the LoggingAlphaOptions feature gate to use this.--logging-format string Default: "text"Sets the log format. Permitted formats: "text".--master stringThe address of the Kubernetes API server (overrides any value in kubeconfig)--permit-address-sharingIf true, SO_REUSEADDR will be used when binding the port. This allows binding to wildcard IPs like 0.0.0.0 and specific IPs in parallel, and it avoids waiting for the kernel to release sockets in TIME_WAIT state. [default=false]--permit-port-sharingIf true, SO_REUSEPORT will be used when binding the port, which allows more than one instance to bind on the same address and port. [default=false]--pod-max-in-unschedulable-pods-duration duration Default: 5m0sDEPRECATED: the maximum time a pod can stay in unschedulablePods. If a pod stays in unschedulablePods for longer than this value, the pod will be moved from unschedulablePods to backoffQ or activeQ. This flag is deprecated and will be removed in a future version.--profiling Default: trueDEPRECATED: enable profiling via web interface host:port/debug/pprof/. This parameter is ignored if a config file is specified in --config.--requestheader-allowed-names stringsList of client certificate common names to allow to provide usernames in headers specified by --requestheader-username-headers. If empty, any client certificate validated by the authorities in --requestheader-client-ca-file is allowed.--requestheader-client-ca-file stringRoot certificate bundle to use to verify client certificates on incoming requests before trusting usernames in headers specified by --requestheader-username-headers. WARNING: generally do not depend on authorization being already done for incoming requests.--requestheader-extra-headers-prefix strings Default: "x-remote-extra-"List of request header prefixes to inspect. X-Remote-Extra- is suggested.--requestheader-group-headers strings Default: "x-remote-group"List of request headers to inspect for groups. X-Remote-Group is suggested.--requestheader-uid-headers stringsList of request headers to inspect for UIDs. X-Remote-Uid is suggested. Requires the RemoteRequestHeaderUID feature to be enabled.--requestheader-username-headers strings Default: "x-remote-user"List of request headers to inspect for usernames. X-Remote-User is common.--secure-port int Default: 10259The port on which to serve HTTPS with authentication and authorization. If 0, don't serve HTTPS at all.--show-hidden-metrics-for-version stringThe previous version for which you want to show hidden metrics. Only the previous minor version is meaningful, other values will not be allowed. The format is <major>.<minor>, e.g.: '1.16'. The purpose of this format is make sure you have the opportunity to notice if the next release hides additional metrics, rather than being surprised when they are permanently removed in the release after that.--tls-cert-file stringFile containing the default x509 Certificate for HTTPS. (CA cert, if any, concatenated after server cert). If HTTPS serving is enabled, and --tls-cert-file and --tls-private-key-file are not provided, a self-signed certificate and key are generated for the public address and saved to the directory specified by --cert-dir.--tls-cipher-suites stringsComma-separated list of cipher suites for the server. If omitted, the default Go cipher suites will be used.Preferred values: TLS_AES_128_GCM_SHA256, TLS_AES_256_GCM_SHA384, TLS_CHACHA20_POLY1305_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305, TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305, TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256.Insecure values: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_RC4_128_SHA, TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_RC4_128_SHA, TLS_RSA_WITH_3DES_EDE_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_256_GCM_SHA384, TLS_RSA_WITH_RC4_128_SHA.--tls-min-version stringMinimum TLS version supported. Possible values: VersionTLS10, VersionTLS11, VersionTLS12, VersionTLS13--tls-private-key-file stringFile containing the default x509 private key matching --tls-cert-file.--tls-sni-cert-key stringA pair of x509 certificate and private key file paths, optionally suffixed with a list of domain patterns which are fully qualified domain names, possibly with prefixed wildcard segments. The domain patterns also allow IP addresses, but IPs should only be used if the apiserver has visibility to the IP address requested by a client. If no domain patterns are provided, the names of the certificate are extracted. Non-wildcard matches trump over wildcard matches, explicit domain patterns trump over extracted names. For multiple key/certificate pairs, use the --tls-sni-cert-key multiple times. Examples: "example.crt,example.key" or "foo.crt,foo.key:*.foo.com,foo.com".-v, --v intnumber for the log level verbosity--version version[=true]--version, --version=raw prints version information and quits; --version=vX.Y.Z... sets the reported version--vmodule pattern=N,...comma-separated list of pattern=N settings for file-filtered logging (only works for text log format)--write-config-to stringIf set, write the configuration values to this file and exit.This page is automatically generated.If you plan to report an issue with this page, mention that the page is auto-generated in your issue description. The fix may need to happen elsewhere in the Kubernetes project.

kube-scheduler


Use Cascading Deletion in a ClusterThis page shows you how to specify the type of cascading deletion to use in your cluster during garbage collection.Before you beginYou need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. It is recommended to run this tutorial on a cluster with at least two nodes that are not acting as control plane hosts. If you do not already have a cluster, you can create one by using minikube or you can use one of these Kubernetes playgrounds:iximiuz LabsKillercodaKodeKloudPlay with KubernetesYou also need to create a sample Deployment to experiment with the different types of cascading deletion. You will need to recreate the Deployment for each type.Check owner references on your podsCheck that the ownerReferences field is present on your pods:kubectl get pods -l app=nginx --output=yaml The output has an ownerReferences field similar to this:apiVersion: v1 ... ownerReferences: - apiVersion: apps/v1 blockOwnerDeletion: true controller: true kind: ReplicaSet name: nginx-deployment-6b474476c4 uid: 4fdcd81c-bd5d-41f7-97af-3a3b759af9a7 ... Use foreground cascading deletionBy default, Kubernetes uses background cascading deletion to delete dependents of an object. You can switch to foreground cascading deletion using either kubectl or the Kubernetes API, depending on the Kubernetes version your cluster runs.To check the version, enter kubectl version.You can delete objects using foreground cascading deletion using kubectl or the Kubernetes API.Using kubectlRun the following command:kubectl delete deployment nginx-deployment --cascade=foreground Using the Kubernetes APIStart a local proxy session:kubectl proxy --port=8080 Use curl to trigger deletion:curl -X DELETE localhost:8080/apis/apps/v1/namespaces/default/deployments/nginx-deployment \ -d '{"kind":"DeleteOptions","apiVersion":"v1","propagationPolicy":"Foreground"}' \ -H "Content-Type: application/json" The output contains a foregroundDeletion finalizer like this:"kind": "Deployment", "apiVersion": "apps/v1", "metadata": { "name": "nginx-deployment", "namespace": "default", "uid": "d1ce1b02-cae8-4288-8a53-30e84d8fa505", "resourceVersion": "1363097", "creationTimestamp": "2021-07-08T20:24:37Z", "deletionTimestamp": "2021-07-08T20:27:39Z", "finalizers": [ "foregroundDeletion" ] ... Use background cascading deletionCreate a sample Deployment.Use either kubectl or the Kubernetes API to delete the Deployment, depending on the Kubernetes version your cluster runs.To check the version, enter kubectl version.You can delete objects using background cascading deletion using kubectl or the Kubernetes API.Kubernetes uses background cascading deletion by default, and does so even if you run the following commands without the --cascade flag or the propagationPolicy argument.Using kubectlRun the following command:kubectl delete deployment nginx-deployment --cascade=background Using the Kubernetes APIStart a local proxy session:kubectl proxy --port=8080 Use curl to trigger deletion:curl -X DELETE localhost:8080/apis/apps/v1/namespaces/default/deployments/nginx-deployment \ -d '{"kind":"DeleteOptions","apiVersion":"v1","propagationPolicy":"Background"}' \ -H "Content-Type: application/json" The output is similar to this:"kind": "Status", "apiVersion": "v1", ... "status": "Success", "details": { "name": "nginx-deployment", "group": "apps", "kind": "deployments", "uid": "cc9eefb9-2d49-4445-b1c1-d261c9396456" } Delete owner objects and orphan dependentsBy default, when you tell Kubernetes to delete an object, the controller also deletes dependent objects. You can make Kubernetes orphan these dependents using kubectl or the Kubernetes API, depending on the Kubernetes version your cluster runs.To check the version, enter kubectl version.Using kubectlRun the following command:kubectl delete deployment nginx-deployment --cascade=orphan Using the Kubernetes APIStart a local proxy session:kubectl proxy --port=8080 Use curl to trigger deletion:curl -X DELETE localhost:8080/apis/apps/v1/namespaces/default/deployments/nginx-deployment \ -d '{"kind":"DeleteOptions","apiVersion":"v1","propagationPolicy":"Orphan"}' \ -H "Content-Type: application/json" The output contains orphan in the finalizers field, similar to this:"kind": "Deployment", "apiVersion": "apps/v1", "namespace": "default", "uid": "6f577034-42a0-479d-be21-78018c466f1f", "creationTimestamp": "2021-07-09T16:46:37Z", "deletionTimestamp": "2021-07-09T16:47:08Z", "deletionGracePeriodSeconds": 0, "finalizers": [ "orphan" ], ... You can check that the Pods managed by the Deployment are still running:kubectl get pods -l app=nginx What's nextLearn about owners and dependents in Kubernetes.Learn about Kubernetes finalizers.Learn about garbage collection.

Use Cascading Deletion in a Cluster


Migrate Replicated Control Plane To Use Cloud Controller ManagerThe cloud-controller-manager is a Kubernetes control plane component that embeds cloud-specific control logic. The cloud controller manager lets you link your cluster into your cloud provider's API, and separates out the components that interact with that cloud platform from components that only interact with your cluster.By decoupling the interoperability logic between Kubernetes and the underlying cloud infrastructure, the cloud-controller-manager component enables cloud providers to release features at a different pace compared to the main Kubernetes project.BackgroundAs part of the cloud provider extraction effort, all cloud specific controllers must be moved out of the kube-controller-manager. All existing clusters that run cloud controllers in the kube-controller-manager must migrate to instead run the controllers in a cloud provider specific cloud-controller-manager.Leader Migration provides a mechanism in which HA clusters can safely migrate "cloud specific" controllers between the kube-controller-manager and the cloud-controller-manager via a shared resource lock between the two components while upgrading the replicated control plane. For a single-node control plane, or if unavailability of controller managers can be tolerated during the upgrade, Leader Migration is not needed and this guide can be ignored.Leader Migration can be enabled by setting --enable-leader-migration on kube-controller-manager or cloud-controller-manager. Leader Migration only applies during the upgrade and can be safely disabled or left enabled after the upgrade is complete.This guide walks you through the manual process of upgrading the control plane from kube-controller-manager with built-in cloud provider to running both kube-controller-manager and cloud-controller-manager. If you use a tool to deploy and manage the cluster, please refer to the documentation of the tool and the cloud provider for specific instructions of the migration.Before you beginIt is assumed that the control plane is running Kubernetes version N and to be upgraded to version N + 1. Although it is possible to migrate within the same version, ideally the migration should be performed as part of an upgrade so that changes of configuration can be aligned to each release. The exact versions of N and N + 1 depend on each cloud provider. For example, if a cloud provider builds a cloud-controller-manager to work with Kubernetes 1.24, then N can be 1.23 and N + 1 can be 1.24.The control plane nodes should run kube-controller-manager with Leader Election enabled, which is the default. As of version N, an in-tree cloud provider must be set with --cloud-provider flag and cloud-controller-manager should not yet be deployed.The out-of-tree cloud provider must have built a cloud-controller-manager with Leader Migration implementation. If the cloud provider imports k8s.io/cloud-provider and k8s.io/controller-manager of version v0.21.0 or later, Leader Migration will be available. However, for version before v0.22.0, Leader Migration is alpha and requires feature gate ControllerManagerLeaderMigration to be enabled in cloud-controller-manager.This guide assumes that kubelet of each control plane node starts kube-controller-manager and cloud-controller-manager as static pods defined by their manifests. If the components run in a different setting, please adjust the steps accordingly.For authorization, this guide assumes that the cluster uses RBAC. If another authorization mode grants permissions to kube-controller-manager and cloud-controller-manager components, please grant the needed access in a way that matches the mode.Grant access to Migration LeaseThe default permissions of the controller manager allow only accesses to their main Lease. In order for the migration to work, accesses to another Lease are required.You can grant kube-controller-manager full access to the leases API by modifying the system::leader-locking-kube-controller-manager role. This task guide assumes that the name of the migration lease is cloud-provider-extraction-migration.kubectl patch -n kube-system role 'system::leader-locking-kube-controller-manager' -p '{"rules": [ {"apiGroups":[ "coordination.k8s.io"], "resources": ["leases"], "resourceNames": ["cloud-provider-extraction-migration"], "verbs": ["create", "list", "get", "update"] } ]}' --type=merge` Do the same to the system::leader-locking-cloud-controller-manager role.kubectl patch -n kube-system role 'system::leader-locking-cloud-controller-manager' -p '{"rules": [ {"apiGroups":[ "coordination.k8s.io"], "resources": ["leases"], "resourceNames": ["cloud-provider-extraction-migration"], "verbs": ["create", "list", "get", "update"] } ]}' --type=merge` Initial Leader Migration configurationLeader Migration optionally takes a configuration file representing the state of controller-to-manager assignment. At this moment, with in-tree cloud provider, kube-controller-manager runs route, service, and cloud-node-lifecycle. The following example configuration shows the assignment.Leader Migration can be enabled without a configuration. Please see Default Configuration for details.kind: LeaderMigrationConfiguration apiVersion: controllermanager.config.k8s.io/v1 leaderName: cloud-provider-extraction-migration resourceLock: leases controllerLeaders: - name: route component: kube-controller-manager - name: service component: kube-controller-manager - name: cloud-node-lifecycle component: kube-controller-manager Alternatively, because the controllers can run under either controller managers, setting component to * for both sides makes the configuration file consistent between both parties of the migration.# wildcard version kind: LeaderMigrationConfiguration apiVersion: controllermanager.config.k8s.io/v1 leaderName: cloud-provider-extraction-migration resourceLock: leases controllerLeaders: - name: route component: * - name: service component: * - name: cloud-node-lifecycle component: * On each control plane node, save the content to /etc/leadermigration.conf, and update the manifest of kube-controller-manager so that the file is mounted inside the container at the same location. Also, update the same manifest to add the following arguments:--enable-leader-migration to enable Leader Migration on the controller manager--leader-migration-config=/etc/leadermigration.conf to set configuration fileRestart kube-controller-manager on each node. At this moment, kube-controller-manager has leader migration enabled and is ready for the migration.Deploy Cloud Controller ManagerIn version N + 1, the desired state of controller-to-manager assignment can be represented by a new configuration file, shown as follows. Please note component field of each controllerLeaders changing from kube-controller-manager to cloud-controller-manager. Alternatively, use the wildcard version mentioned above, which has the same effect.kind: LeaderMigrationConfiguration apiVersion: controllermanager.config.k8s.io/v1 leaderName: cloud-provider-extraction-migration resourceLock: leases controllerLeaders: - name: route component: cloud-controller-manager - name: service component: cloud-controller-manager - name: cloud-node-lifecycle component: cloud-controller-manager When creating control plane nodes of version N + 1, the content should be deployed to /etc/leadermigration.conf. The manifest of cloud-controller-manager should be updated to mount the configuration file in the same manner as kube-controller-manager of version N. Similarly, add --enable-leader-migration and --leader-migration-config=/etc/leadermigration.conf to the arguments of cloud-controller-manager.Create a new control plane node of version N + 1 with the updated cloud-controller-manager manifest, and with the --cloud-provider flag set to external for kube-controller-manager. kube-controller-manager of version N + 1 MUST NOT have Leader Migration enabled because, with an external cloud provider, it does not run the migrated controllers anymore, and thus it is not involved in the migration.Please refer to Cloud Controller Manager Administration for more detail on how to deploy cloud-controller-manager.Upgrade Control PlaneThe control plane now contains nodes of both version N and N + 1. The nodes of version N run kube-controller-manager only, and these of version N + 1 run both kube-controller-manager and cloud-controller-manager. The migrated controllers, as specified in the configuration, are running under either kube-controller-manager of version N or cloud-controller-manager of version N + 1 depending on which controller manager holds the migration lease. No controller will ever be running under both controller managers at any time.In a rolling manner, create a new control plane node of version N + 1 and bring down one of version N until the control plane contains only nodes of version N + 1. If a rollback from version N + 1 to N is required, add nodes of version N with Leader Migration enabled for kube-controller-manager back to the control plane, replacing one of version N + 1 each time until there are only nodes of version N.(Optional) Disable Leader MigrationNow that the control plane has been upgraded to run both kube-controller-manager and cloud-controller-manager of version N + 1, Leader Migration has finished its job and can be safely disabled to save one Lease resource. It is safe to re-enable Leader Migration for the rollback in the future.In a rolling manager, update manifest of cloud-controller-manager to unset both --enable-leader-migration and --leader-migration-config= flag, also remove the mount of /etc/leadermigration.conf, and finally remove /etc/leadermigration.conf. To re-enable Leader Migration, recreate the configuration file and add its mount and the flags that enable Leader Migration back to cloud-controller-manager.Default ConfigurationStarting Kubernetes 1.22, Leader Migration provides a default configuration suitable for the default controller-to-manager assignment. The default configuration can be enabled by setting --enable-leader-migration but without --leader-migration-config=.For kube-controller-manager and cloud-controller-manager, if there are no flags that enable any in-tree cloud provider or change ownership of controllers, the default configuration can be used to avoid manual creation of the configuration file.Special case: migrating the Node IPAM controllerIf your cloud provider provides an implementation of Node IPAM controller, you should switch to the implementation in cloud-controller-manager. Disable Node IPAM controller in kube-controller-manager of version N + 1 by adding --controllers=*,-nodeipam to its flags. Then add nodeipam to the list of migrated controllers.# wildcard version, with nodeipam kind: LeaderMigrationConfiguration apiVersion: controllermanager.config.k8s.io/v1 leaderName: cloud-provider-extraction-migration resourceLock: leases controllerLeaders: - name: route component: * - name: service component: * - name: cloud-node-lifecycle component: * - name: nodeipam - component: * What's nextRead the Controller Manager Leader Migration enhancement proposal.


Client LibrariesThis page contains an overview of the client libraries for using the Kubernetes API from various programming languages.To write applications using the Kubernetes REST API, you do not need to implement the API calls and request/response types yourself. You can use a client library for the programming language you are using.Client libraries often handle common tasks such as authentication for you. Most client libraries can discover and use the Kubernetes Service Account to authenticate if the API client is running inside the Kubernetes cluster, or can understand the kubeconfig file format to read the credentials and the API Server address.Officially-supported Kubernetes client librariesThe following client libraries are officially maintained by Kubernetes SIG API Machinery.LanguageClient LibrarySample ProgramsCgithub.com/kubernetes-client/cbrowsedotnetgithub.com/kubernetes-client/csharpbrowseGogithub.com/kubernetes/client-go/browseHaskellgithub.com/kubernetes-client/haskellbrowseJavagithub.com/kubernetes-client/javabrowseJavaScriptgithub.com/kubernetes-client/javascriptbrowsePerlgithub.com/kubernetes-client/perl/browsePythongithub.com/kubernetes-client/python/browseRubygithub.com/kubernetes-client/ruby/browseCommunity-maintained client librariesNote: This section links to third party projects that provide functionality required by Kubernetes. The Kubernetes project authors aren't responsible for these projects, which are listed alphabetically. To add a project to this list, read the content guide before submitting a change. More information.The following Kubernetes API client libraries are provided and maintained by their authors, not the Kubernetes team.LanguageClient LibraryClojuregithub.com/yanatan16/clj-kubernetes-apiDotNetgithub.com/tonnyeremin/kubernetes_genDotNet (RestSharp)github.com/masroorhasan/Kubernetes.DotNetElixirgithub.com/obmarg/kazanElixirgithub.com/coryodaniel/k8sJava (OSGi)bitbucket.org/amdatulabs/amdatu-kubernetesJava (Fabric8, OSGi)github.com/fabric8io/kubernetes-clientJavagithub.com/manusa/yakcLispgithub.com/brendandburns/cl-k8sLispgithub.com/xh4/cubeNode.js (TypeScript)github.com/Goyoo/node-k8s-clientNode.jsgithub.com/ajpauwels/easy-k8sNode.jsgithub.com/godaddy/kubernetes-clientNode.jsgithub.com/tenxcloud/node-kubernetes-clientPerlmetacpan.org/pod/Net::KubernetesPHPgithub.com/allansun/kubernetes-php-clientPHPgithub.com/maclof/kubernetes-clientPHPgithub.com/travisghansen/kubernetes-client-phpPHPgithub.com/renoki-co/php-k8sPythongithub.com/cloudcoil/cloudcoilPythongithub.com/fiaas/k8sPythongithub.com/gtsystem/lightkubePythongithub.com/kr8s-org/kr8sPythongithub.com/mnubo/kubernetes-pyPythongithub.com/tomplus/kubernetes_asyncioPythongithub.com/Frankkkkk/pykormRubygithub.com/abonas/kubeclientRubygithub.com/k8s-ruby/k8s-rubyRubygithub.com/kontena/k8s-clientRustgithub.com/kube-rs/kubeRustgithub.com/ynqa/kubernetes-rustScalagithub.com/hagay3/skuberScalagithub.com/hnaderi/scala-k8sScalagithub.com/joan38/kubernetes-clientSwiftgithub.com/swiftkube/clientItems on this page refer to third party products or projects that provide functionality required by Kubernetes. The Kubernetes project authors aren't responsible for those third-party products or projects. See the CNCF website guidelines for more details.You should read the content guide before proposing a change that adds an extra third-party link.

Client Libraries


Configure Default CPU Requests and Limits for a NamespaceDefine a default CPU resource limits for a namespace, so that every new Pod in that namespace has a CPU resource limit configured.This page shows how to configure default CPU requests and limits for a namespace.A Kubernetes cluster can be divided into namespaces. If you create a Pod within a namespace that has a default CPU limit, and any container in that Pod does not specify its own CPU limit, then the control plane assigns the default CPU limit to that container.Kubernetes assigns a default CPU request, but only under certain conditions that are explained later in this page.Before you beginYou need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. It is recommended to run this tutorial on a cluster with at least two nodes that are not acting as control plane hosts. If you do not already have a cluster, you can create one by using minikube or you can use one of these Kubernetes playgrounds:iximiuz LabsKillercodaKodeKloudPlay with KubernetesYou must have access to create namespaces in your cluster.If you're not already familiar with what Kubernetes means by 1.0 CPU, read meaning of CPU.Create a namespaceCreate a namespace so that the resources you create in this exercise are isolated from the rest of your cluster.kubectl create namespace default-cpu-example Create a LimitRange and a PodHere's a manifest for an example LimitRange. The manifest specifies a default CPU request and a default CPU limit.admin/resource/cpu-defaults.yaml apiVersion: v1 kind: LimitRange metadata: name: cpu-limit-range spec: limits: - default: cpu: 1 defaultRequest: cpu: 0.5 type: Container Create the LimitRange in the default-cpu-example namespace:kubectl apply -f https://k8s.io/examples/admin/resource/cpu-defaults.yaml --namespace=default-cpu-example Now if you create a Pod in the default-cpu-example namespace, and any container in that Pod does not specify its own values for CPU request and CPU limit, then the control plane applies default values: a CPU request of 0.5 and a default CPU limit of 1.Here's a manifest for a Pod that has one container. The container does not specify a CPU request and limit.admin/resource/cpu-defaults-pod.yaml apiVersion: v1 kind: Pod metadata: name: default-cpu-demo spec: containers: - name: default-cpu-demo-ctr image: nginx Create the Pod.kubectl apply -f https://k8s.io/examples/admin/resource/cpu-defaults-pod.yaml --namespace=default-cpu-example View the Pod's specification:kubectl get pod default-cpu-demo --output=yaml --namespace=default-cpu-example The output shows that the Pod's only container has a CPU request of 500m cpu (which you can read as “500 millicpu”), and a CPU limit of 1 cpu. These are the default values specified by the LimitRange.containers: - image: nginx imagePullPolicy: Always name: default-cpu-demo-ctr resources: limits: cpu: "1" requests: cpu: 500m What if you specify a container's limit, but not its request?Here's a manifest for a Pod that has one container. The container specifies a CPU limit, but not a request:admin/resource/cpu-defaults-pod-2.yaml apiVersion: v1 kind: Pod metadata: name: default-cpu-demo-2 spec: containers: - name: default-cpu-demo-2-ctr image: nginx resources: limits: cpu: "1" Create the Pod:kubectl apply -f https://k8s.io/examples/admin/resource/cpu-defaults-pod-2.yaml --namespace=default-cpu-example View the specification of the Pod that you created:kubectl get pod default-cpu-demo-2 --output=yaml --namespace=default-cpu-example The output shows that the container's CPU request is set to match its CPU limit. Notice that the container was not assigned the default CPU request value of 0.5 cpu:resources: limits: cpu: "1" requests: cpu: "1" What if you specify a container's request, but not its limit?Here's an example manifest for a Pod that has one container. The container specifies a CPU request, but not a limit:admin/resource/cpu-defaults-pod-3.yaml apiVersion: v1 kind: Pod metadata: name: default-cpu-demo-3 spec: containers: - name: default-cpu-demo-3-ctr image: nginx resources: requests: cpu: "0.75" Create the Pod:kubectl apply -f https://k8s.io/examples/admin/resource/cpu-defaults-pod-3.yaml --namespace=default-cpu-example View the specification of the Pod that you created:kubectl get pod default-cpu-demo-3 --output=yaml --namespace=default-cpu-example The output shows that the container's CPU request is set to the value you specified at the time you created the Pod (in other words: it matches the manifest). However, the same container's CPU limit is set to 1 cpu, which is the default CPU limit for that namespace.resources: limits: cpu: "1" requests: cpu: 750m Motivation for default CPU limits and requestsIf your namespace has a CPU resource quota configured, it is helpful to have a default value in place for CPU limit. Here are two of the restrictions that a CPU resource quota imposes on a namespace:For every Pod that runs in the namespace, each of its containers must have a CPU limit.CPU limits apply a resource reservation on the node where the Pod in question is scheduled. The total amount of CPU that is reserved for use by all Pods in the namespace must not exceed a specified limit.When you add a LimitRange:If any Pod in that namespace that includes a container does not specify its own CPU limit, the control plane applies the default CPU limit to that container, and the Pod can be allowed to run in a namespace that is restricted by a CPU ResourceQuota.Clean upDelete your namespace:kubectl delete namespace default-cpu-example What's nextFor cluster administratorsConfigure Default Memory Requests and Limits for a NamespaceConfigure Minimum and Maximum Memory Constraints for a NamespaceConfigure Minimum and Maximum CPU Constraints for a NamespaceConfigure Memory and CPU Quotas for a NamespaceConfigure a Pod Quota for a NamespaceConfigure Quotas for API ObjectsFor app developersAssign Memory Resources to Containers and PodsAssign CPU Resources to Containers and PodsAssign Pod-level CPU and memory resourcesConfigure Quality of Service for Pods


Scheduler Performance TuningFEATURE STATE: Kubernetes v1.14 [beta]kube-scheduler is the Kubernetes default scheduler. It is responsible for placement of Pods on Nodes in a cluster.Nodes in a cluster that meet the scheduling requirements of a Pod are called feasible Nodes for the Pod. The scheduler finds feasible Nodes for a Pod and then runs a set of functions to score the feasible Nodes, picking a Node with the highest score among the feasible ones to run the Pod. The scheduler then notifies the API server about this decision in a process called Binding.This page explains performance tuning optimizations that are relevant for large Kubernetes clusters.In large clusters, you can tune the scheduler's behaviour balancing scheduling outcomes between latency (new Pods are placed quickly) and accuracy (the scheduler rarely makes poor placement decisions).You configure this tuning setting via kube-scheduler setting percentageOfNodesToScore. This KubeSchedulerConfiguration setting determines a threshold for scheduling nodes in your cluster.Setting the thresholdThe percentageOfNodesToScore option accepts whole numeric values between 0 and 100. The value 0 is a special number which indicates that the kube-scheduler should use its compiled-in default. If you set percentageOfNodesToScore above 100, kube-scheduler acts as if you had set a value of 100.To change the value, edit the kube-scheduler configuration file and then restart the scheduler. In many cases, the configuration file can be found at /etc/kubernetes/config/kube-scheduler.yaml.After you have made this change, you can runkubectl get pods -n kube-system | grep kube-scheduler to verify that the kube-scheduler component is healthy.Node scoring thresholdTo improve scheduling performance, the kube-scheduler can stop looking for feasible nodes once it has found enough of them. In large clusters, this saves time compared to a naive approach that would consider every node.You specify a threshold for how many nodes are enough, as a whole number percentage of all the nodes in your cluster. The kube-scheduler converts this into an integer number of nodes. During scheduling, if the kube-scheduler has identified enough feasible nodes to exceed the configured percentage, the kube-scheduler stops searching for more feasible nodes and moves on to the scoring phase.How the scheduler iterates over Nodes describes the process in detail.Default thresholdIf you don't specify a threshold, Kubernetes calculates a figure using a linear formula that yields 50% for a 100-node cluster and yields 10% for a 5000-node cluster. The lower bound for the automatic value is 5%.This means that the kube-scheduler always scores at least 5% of your cluster no matter how large the cluster is, unless you have explicitly set percentageOfNodesToScore to be smaller than 5.If you want the scheduler to score all nodes in your cluster, set percentageOfNodesToScore to 100.ExampleBelow is an example configuration that sets percentageOfNodesToScore to 50%.apiVersion: kubescheduler.config.k8s.io/v1alpha1 kind: KubeSchedulerConfiguration algorithmSource: provider: DefaultProvider ... percentageOfNodesToScore: 50 Tuning percentageOfNodesToScorepercentageOfNodesToScore must be a value between 1 and 100 with the default value being calculated based on the cluster size. There is also a hardcoded minimum value of 100 nodes.Note:In clusters with less than 100 feasible nodes, the scheduler still checks all the nodes because there are not enough feasible nodes to stop the scheduler's search early.In a small cluster, if you set a low value for percentageOfNodesToScore, your change will have no or little effect, for a similar reason.If your cluster has several hundred Nodes or fewer, leave this configuration option at its default value. Making changes is unlikely to improve the scheduler's performance significantly.An important detail to consider when setting this value is that when a smaller number of nodes in a cluster are checked for feasibility, some nodes are not sent to be scored for a given Pod. As a result, a Node which could possibly score a higher value for running the given Pod might not even be passed to the scoring phase. This would result in a less than ideal placement of the Pod.You should avoid setting percentageOfNodesToScore very low so that kube-scheduler does not make frequent, poor Pod placement decisions. Avoid setting the percentage to anything below 10%, unless the scheduler's throughput is critical for your application and the score of nodes is not important. In other words, you prefer to run the Pod on any Node as long as it is feasible.How the scheduler iterates over NodesThis section is intended for those who want to understand the internal details of this feature.In order to give all the Nodes in a cluster a fair chance of being considered for running Pods, the scheduler iterates over the nodes in a round robin fashion. You can imagine that Nodes are in an array. The scheduler starts from the start of the array and checks feasibility of the nodes until it finds enough Nodes as specified by percentageOfNodesToScore. For the

Scheduler Performance Tuning


Building a Basic DaemonSetThis page demonstrates how to build a basic DaemonSet that runs a Pod on every node in a Kubernetes cluster. It covers a simple use case of mounting a file from the host, logging its contents using an init container, and utilizing a pause container.Before you beginYou need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. It is recommended to run this tutorial on a cluster with at least two nodes that are not acting as control plane hosts. If you do not already have a cluster, you can create one by using minikube or you can use one of these Kubernetes playgrounds:iximiuz LabsKillercodaKodeKloudPlay with KubernetesA Kubernetes cluster with at least two nodes (one control plane node and one worker node) to demonstrate the behavior of DaemonSets.Define the DaemonSetIn this task, a basic DaemonSet is created which ensures that the copy of a Pod is scheduled on every node. The Pod will use an init container to read and log the contents of /etc/machine-id from the host, while the main container will be a pause container, which keeps the Pod running.application/basic-daemonset.yaml apiVersion: apps/v1 kind: DaemonSet metadata: name: example-daemonset spec: selector: matchLabels: app.kubernetes.io/name: example template: metadata: labels: app.kubernetes.io/name: example spec: containers: - name: pause image: registry.k8s.io/pause initContainers: - name: log-machine-id image: busybox:1.37 command: ['sh', '-c', 'cat /etc/machine-id > /var/log/machine-id.log'] volumeMounts: - name: machine-id mountPath: /etc/machine-id readOnly: true - name: log-dir mountPath: /var/log volumes: - name: machine-id hostPath: path: /etc/machine-id type: File - name: log-dir hostPath: path: /var/logCreate a DaemonSet based on the (YAML) manifest:kubectl apply -f https://k8s.io/examples/application/basic-daemonset.yaml Once applied, you can verify that the DaemonSet is running a Pod on every node in the cluster:kubectl get pods -o wide The output will list one Pod per node, similar to:NAME READY STATUS RESTARTS AGE IP NODE example-daemonset-xxxxx 1/1 Running 0 5m x.x.x.x node-1 example-daemonset-yyyyy 1/1 Running 0 5m x.x.x.x node-2 You can inspect the contents of the logged /etc/machine-id file by checking the log directory mounted from the host:kubectl exec <pod-name> -- cat /var/log/machine-id.log Where <pod-name> is the name of one of your Pods.Cleaning upTo delete the DaemonSet, run this command:kubectl delete --cascade=foreground --ignore-not-found --now daemonsets/example-daemonset This simple DaemonSet example introduces key components like init containers and host path volumes, which can be expanded upon for more advanced use cases. For more details refer to DaemonSet.What's nextSee Performing a rolling update on a DaemonSetSee Creating a DaemonSet to adopt existing DaemonSet pods

Building a Basic DaemonSet


Issue WranglersAlongside the PR Wrangler, formal approvers, reviewers and members of SIG Docs take week-long shifts triaging and categorising issues for the repository.DutiesEach day in a week-long shift the Issue Wrangler will be responsible for:Triaging and tagging incoming issues daily. See Triage and categorize issues for guidelines on how SIG Docs uses metadata.Keeping an eye on stale & rotten issues within the kubernetes/website repository.Maintenance of the Issues board.RequirementsMust be an active member of the Kubernetes organization.A minimum of 15 non-trivial contributions to Kubernetes (of which a certain amount should be directed towards kubernetes/website).Performing the role in an informal capacity already.Helpful Prow commands for wranglersBelow are some commonly used commands for Issue Wranglers:# reopen an issue /reopen # transfer issues that don't fit in k/website to another repository /transfer[-issue] # change the state of rotten issues /remove-lifecycle rotten # change the state of stale issues /remove-lifecycle stale # assign sig to an issue /sig <sig_name> # add specific area /area <area_name> # for beginner friendly issues /good-first-issue # issues that needs help /help wanted # tagging issue as support specific /kind support # to accept triaging for an issue /triage accepted # closing an issue we won't be working on and haven't fixed yet /close not-planned To find more Prow commands, refer to the Command Help documentation.When to close IssuesFor an open source project to succeed, good issue management is crucial. But it is also critical to resolve issues in order to maintain the repository and communicate clearly with contributors and users.Close issues when:A similar issue is reported more than once. You will first need to tag it as /triage duplicate; link it to the main issue & then close it. It is also advisable to direct the users to the original issue.It is very difficult to understand and address the issue presented by the author with the information provided. However, encourage the user to provide more details or reopen the issue if they can reproduce it later.The same functionality is implemented elsewhere. One can close this issue and direct user to the appropriate place.The reported issue is not currently planned or aligned with the project's goals.If the issue appears to be spam and is clearly unrelated.If the issue is related to an external limitation or dependency and is beyond the control of the project.To close an issue, leave a /close comment on the issue.

Issue Wranglers


Options for Highly Available TopologyThis page explains the two options for configuring the topology of your highly available (HA) Kubernetes clusters.You can set up an HA cluster:With stacked control plane nodes, where etcd nodes are colocated with control plane nodesWith external etcd nodes, where etcd runs on separate nodes from the control planeYou should carefully consider the advantages and disadvantages of each topology before setting up an HA cluster.Note:kubeadm bootstraps the etcd cluster statically. Read the etcd Clustering Guide for more details.Stacked etcd topologyA stacked HA cluster is a topology where the distributed data storage cluster provided by etcd is stacked on top of the cluster formed by the nodes managed by kubeadm that run control plane components.Each control plane node runs an instance of the kube-apiserver, kube-scheduler, and kube-controller-manager. The kube-apiserver is exposed to worker nodes using a load balancer.Each control plane node creates a local etcd member and this etcd member communicates only with the kube-apiserver of this node. The same applies to the local kube-controller-manager and kube-scheduler instances.This topology couples the control planes and etcd members on the same nodes. It is simpler to set up than a cluster with external etcd nodes, and simpler to manage for replication.However, a stacked cluster runs the risk of failed coupling. If one node goes down, both an etcd member and a control plane instance are lost, and redundancy is compromised. You can mitigate this risk by adding more control plane nodes.You should therefore run a minimum of three stacked control plane nodes for an HA cluster.This is the default topology in kubeadm. A local etcd member is created automatically on control plane nodes when using kubeadm init and kubeadm join --control-plane.External etcd topologyAn HA cluster with external etcd is a topology where the distributed data storage cluster provided by etcd is external to the cluster formed by the nodes that run control plane components.Like the stacked etcd topology, each control plane node in an external etcd topology runs an instance of the kube-apiserver, kube-scheduler, and kube-controller-manager. And the kube-apiserver is exposed to worker nodes using a load balancer. However, etcd members run on separate hosts, and each etcd host communicates with the kube-apiserver of each control plane node.This topology decouples the control plane and etcd member. It therefore provides an HA setup where losing a control plane instance or an etcd member has less impact and does not affect the cluster redundancy as much as the stacked HA topology.However, this topology requires twice the number of hosts as the stacked HA topology. A minimum of three hosts for control plane nodes and three hosts for etcd nodes are required for an HA cluster with this topology.What's nextSet up a highly available cluster with kubeadm

Options for Highly Available Topology


Windows containers in KubernetesWindows applications constitute a large portion of the services and applications that run in many organizations. Windows containers provide a way to encapsulate processes and package dependencies, making it easier to use DevOps practices and follow cloud native patterns for Windows applications.Organizations with investments in Windows-based applications and Linux-based applications don't have to look for separate orchestrators to manage their workloads, leading to increased operational efficiencies across their deployments, regardless of operating system.Windows nodes in KubernetesTo enable the orchestration of Windows containers in Kubernetes, include Windows nodes in your existing Linux cluster. Scheduling Windows containers in Pods on Kubernetes is similar to scheduling Linux-based containers.In order to run Windows containers, your Kubernetes cluster must include multiple operating systems. While you can only run the control plane on Linux, you can deploy worker nodes running either Windows or Linux.Windows nodes are supported provided that the operating system is Windows Server 2019 or Windows Server 2022.This document uses the term Windows containers to mean Windows containers with process isolation. Kubernetes does not support running Windows containers with Hyper-V isolation.Compatibility and limitationsSome node features are only available if you use a specific container runtime; others are not available on Windows nodes, including:HugePages: not supported for Windows containersPrivileged containers: not supported for Windows containers. HostProcess Containers offer similar functionality.TerminationGracePeriod: requires containerDNot all features of shared namespaces are supported. See API compatibility for more details.See Windows OS version compatibility for details on the Windows versions that Kubernetes is tested against.From an API and kubectl perspective, Windows containers behave in much the same way as Linux-based containers. However, there are some notable differences in key functionality which are outlined in this section.Comparison with LinuxKey Kubernetes elements work the same way in Windows as they do in Linux. This section refers to several key workload abstractions and how they map to Windows.PodsA Pod is the basic building block of Kubernetes–the smallest and simplest unit in the Kubernetes object model that you create or deploy. You may not deploy Windows and Linux containers in the same Pod. All containers in a Pod are scheduled onto a single Node where each Node represents a specific platform and architecture. The following Pod capabilities, properties and events are supported with Windows containers:Single or multiple containers per Pod with process isolation and volume sharingPod status fieldsReadiness, liveness, and startup probespostStart & preStop container lifecycle hooksConfigMap, Secrets: as environment variables or volumesemptyDir volumesNamed pipe host mountsResource limitsOS field:The .spec.os.name field should be set to windows to indicate that the current Pod uses Windows containers.If you set the .spec.os.name field to windows, you must not set the following fields in the .spec of that Pod:spec.hostPIDspec.hostIPCspec.securityContext.seLinuxOptionsspec.securityContext.seccompProfilespec.securityContext.fsGroupspec.securityContext.fsGroupChangePolicyspec.securityContext.sysctlsspec.shareProcessNamespacespec.securityContext.runAsUserspec.securityContext.runAsGroupspec.securityContext.supplementalGroupsspec.containers[*].securityContext.seLinuxOptionsspec.containers[*].securityContext.seccompProfilespec.containers[*].securityContext.capabilitiesspec.containers[*].securityContext.readOnlyRootFilesystemspec.containers[*].securityContext.privilegedspec.containers[*].securityContext.allowPrivilegeEscalationspec.containers[*].securityContext.procMountspec.containers[*].securityContext.runAsUserspec.containers[*].securityContext.runAsGroupIn the above list, wildcards (*) indicate all elements in a list. For example, spec.containers[*].securityContext refers to the SecurityContext object for all containers. If any of these fields is specified, the Pod will not be admitted by the API server.Workload resources including:ReplicaSetDeploymentStatefulSetDaemonSetJobCronJobReplicationControllerServices See Load balancing and Services for more details.Pods, workload resources, and Services are critical elements to managing Windows workloads on Kubernetes. However, on their own they are not enough to enable the proper lifecycle management of Windows workloads in a dynamic cloud native environment.kubectl execPod and container metricsHorizontal pod autoscalingResource quotasScheduler preemptionCommand line options for the kubeletSome kubelet command line options behave differently on Windows, as described below:The --windows-priorityclass lets you set the scheduling priority of the kubelet process (see CPU resource management)The --kube-reserved, --system-reserved , and --eviction-hard flags update NodeAllocatableEviction by using --enforce-node-allocable is not implementedWhen running on a Windows node the kubelet does not have memory or CPU restrictions. --kube-reserved and --system-reserved only subtract from NodeAllocatable and do not guarantee resource provided for workloads. See Resource Management for Windows nodes for more information.The PIDPressure Condition is not implementedThe kubelet does not take OOM eviction actionsAPI compatibilityThere are subtle differences in the way the Kubernetes APIs work for Windows due to the OS and container runtime. Some workload properties were designed for Linux, and fail to run on Windows.At a high level, these OS concepts are different:Identity - Linux uses userID (UID) and groupID (GID) which are represented as integer types. User and group names are not canonical - they are just an alias in /etc/groups or /etc/passwd back to UID+GID. Windows uses a larger binary security identifier (SID) which is stored in the Windows Security Access Manager (SAM) database. This database is not shared between the host and containers, or between containers.File permissions - Windows uses an access control list based on (SIDs), whereas POSIX systems such as Linux use a bitmask based on object permissions and UID+GID, plus optional access control lists.File paths - the convention on Windows is to use \ instead of /. The Go IO libraries typically accept both and just make it work, but when you're setting a path or command line that's interpreted inside a container, \ may be needed.Signals - Windows interactive apps handle termination differently, and can implement one or more of these:A UI thread handles well-defined messages including WM_CLOSE.Console apps handle Ctrl-C or Ctrl-break using a Control Handler.Services register a Service Control Handler function that can accept SERVICE_CONTROL_STOP control codes.Container exit codes follow the same convention where 0 is success, and nonzero is failure. The specific error codes may differ across Windows and Linux. However, exit codes passed from the Kubernetes components (kubelet, kube-proxy) are unchanged.Field compatibility for container specificationsThe following list documents differences between how Pod container specifications work between Windows and Linux:Huge pages are not implemented in the Windows container runtime, and are not available. They require asserting a user privilege that's not configurable for containers.requests.cpu and requests.memory - requests are subtracted from node available resources, so they can be used to avoid overprovisioning a node. However, they cannot be used to guarantee resources in an overprovisioned node. They should be applied to all containers as a best practice if the operator wants to avoid overprovisioning entirely.securityContext.allowPrivilegeEscalation - not possible on Windows; none of the capabilities are hooked upsecurityContext.capabilities - POSIX capabilities are not implemented on WindowssecurityContext.privileged - Windows doesn't support privileged containers, use HostProcess Containers insteadsecurityContext.procMount - Windows doesn't have a /proc filesystemsecurityContext.readOnlyRootFilesystem - not possible on Windows; write access is required for registry & system processes to run inside the containersecurityContext.runAsGroup - not possible on Windows as there is no GID supportsecurityContext.runAsNonRoot - this setting will prevent containers from running as ContainerAdministrator which is the closest equivalent to a root user on Windows.securityContext.runAsUser - use runAsUserName insteadsecurityContext.seLinuxOptions - not possible on Windows as SELinux is Linux-specificterminationMessagePath - this has some limitations in that Windows doesn't support mapping single files. The default value is /dev/termination-log, which does work because it does not exist on Windows by default.Field compatibility for Pod specificationsThe following list documents differences between how Pod specifications work between Windows and Linux:hostIPC and hostpid - host namespace sharing is not possible on WindowshostNetwork - host networking is not possible on WindowsdnsPolicy - setting the Pod dnsPolicy to ClusterFirstWithHostNet is not supported on Windows because host networking is not provided. Pods always run with a container network.podSecurityContext see belowshareProcessNamespace - this is a beta feature, and depends on Linux namespaces which are not implemented on Windows. Windows cannot share process namespaces or the container's root filesystem. Only the network can be shared.terminationGracePeriodSeconds - this is not fully implemented in Docker on Windows, see the GitHub issue. The behavior today is that the ENTRYPOINT process is sent CTRL_SHUTDOWN_EVENT, then Windows waits 5 seconds by default, and finally shuts down all processes using the normal Windows shutdown behavior. The 5 second default is actually in the Windows registry inside the container, so it can be overridden when the container is built.volumeDevices - this is a beta feature, and is not implemented on Windows. Windows cannot attach raw block devices to pods.volumesIf you define an emptyDir volume, you cannot set its volume source to memory.You cannot enable mountPropagation for volume mounts as this is not supported on Windows.Host network accessKubernetes v1.26 to v1.32 included alpha support for running Windows Pods in the host's network namespace.Kubernetes v1.34 does not include the WindowsHostNetwork feature gate or support for running Windows Pods in the host's network namespace.Field compatibility for Pod security contextOnly the securityContext.runAsNonRoot and securityContext.windowsOptions from the Pod securityContext fields work on Windows.Node problem detectorThe node problem detector (see Monitor Node Health) has preliminary support for Windows. For more information, visit the project's GitHub page.Pause containerIn a Kubernetes Pod, an infrastructure or “pause” container is first created to host the container. In Linux, the cgroups and namespaces that make up a pod need a process to maintain their continued existence; the pause process provides this. Containers that belong to the same pod, including infrastructure and worker containers, share a common network endpoint (same IPv4 and / or IPv6 address, same network port spaces). Kubernetes uses pause containers to allow for worker containers crashing or restarting without losing any of the networking configuration.Kubernetes maintains a multi-architecture image that includes support for Windows. For Kubernetes v1.34.0 the recommended pause image is registry.k8s.io/pause:3.6. The source code is available on GitHub.Microsoft maintains a different multi-architecture image, with Linux and Windows amd64 support, that you can find as mcr.microsoft.com/oss/kubernetes/pause:3.6. This image is built from the same source as the Kubernetes maintained image but all of the Windows binaries are authenticode signed by Microsoft. The Kubernetes project recommends using the Microsoft maintained image if you are deploying to a production or production-like environment that requires signed binaries.Container runtimesYou need to install a container runtime into each node in the cluster so that Pods can run there.The following container runtimes work with Windows:Note: This section links to third party projects that provide functionality required by Kubernetes. The Kubernetes project authors aren't responsible for these projects, which are listed alphabetically. To add a project to this list, read the content guide before submitting a change. More information.ContainerDFEATURE STATE: Kubernetes v1.20 [stable]You can use ContainerD 1.4.0+ as the container runtime for Kubernetes nodes that run Windows.Learn how to install ContainerD on a Windows node.Note:There is a known limitation when using GMSA with containerd to access Windows network shares, which requires a kernel patch.Mirantis Container RuntimeMirantis Container Runtime (MCR) is available as a container runtime for all Windows Server 2019 and later versions.See Install MCR on Windows Servers for more information.Windows OS version compatibilityOn Windows nodes, strict compatibility rules apply where the host OS version must match the container base image OS version. Only Windows containers with a container operating system of Windows Server 2019 are fully supported.For Kubernetes v1.34, operating system compatibility for Windows nodes (and Pods) is as follows:Windows Server LTSC releaseWindows Server 2019Windows Server 2022Windows Server SAC releaseWindows Server version 20H2The Kubernetes version-skew policy also applies.Hardware recommendations and considerationsNote: This section links to third party projects that provide functionality required by Kubernetes. The Kubernetes project authors aren't responsible for these projects, which are listed alphabetically. To add a project to this list, read the content guide before submitting a change. More information.Note:The following hardware specifications outlined here should be regarded as sensible default values. They are not intended to represent minimum requirements or specific recommendations for production environments. Depending on the requirements for your workload these values may need to be adjusted.64-bit processor 4 CPU cores or more, capable of supporting virtualization8GB or more of RAM50GB or more of free disk spaceRefer to Hardware requirements for Windows Server Microsoft documentation for the most up-to-date information on minimum hardware requirements. For guidance on deciding on resources for production worker nodes refer to Production worker nodes Kubernetes documentation.To optimize system resources, if a graphical user interface is not required, it may be preferable to use a Windows Server OS installation that excludes the Windows Desktop Experience installation option, as this configuration typically frees up more system resources.In assessing disk space for Windows worker nodes, take note that Windows container images are typically larger than Linux container images, with container image sizes ranging from 300MB to over 10GB for a single image. Additionally, take note that the C: drive in Windows containers represents a virtual free size of 20GB by default, which is not the actual consumed space, but rather the disk size for which a single container can grow to occupy when using local storage on the host. See Containers on Windows - Container Storage Documentation for more detail.Getting help and troubleshootingYour main source of help for troubleshooting your Kubernetes cluster should start with the Troubleshooting page.Some additional, Windows-specific troubleshooting help is included in this section. Logs are an important element of troubleshooting issues in Kubernetes. Make sure to include them any time you seek troubleshooting assistance from other contributors. Follow the instructions in the SIG Windows contributing guide on gathering logs.Reporting issues and feature requestsIf you have what looks like a bug, or you would like to make a feature request, please follow the SIG Windows contributing guide to create a new issue. You should first search the list of issues in case it was reported previously and comment with your experience on the issue and add additional logs. SIG Windows channel on the Kubernetes Slack is also a great avenue to get some initial support and troubleshooting ideas prior to creating a ticket.Validating the Windows cluster operabilityThe Kubernetes project provides a Windows Operational Readiness specification, accompanied by a structured test suite. This suite is split into two sets of tests, core and extended, each containing categories aimed at testing specific areas. It can be used to validate all the functionalities of a Windows and hybrid system (mixed with Linux nodes) with full coverage.To set up the project on a newly created cluster, refer to the instructions in the project guide.Deployment toolsThe kubeadm tool helps you to deploy a Kubernetes cluster, providing the control plane to manage the cluster it, and nodes to run your workloads.The Kubernetes cluster API project also provides means to automate deployment of Windows nodes.Windows distribution channelsFor a detailed explanation of Windows distribution channels see the Microsoft documentation.Information on the different Windows Server servicing channels including their support models can be found at Windows Server servicing channels.Items on this page refer to third party products or projects that provide functionality required by Kubernetes. The Kubernetes project authors aren't responsible for those third-party products or projects. See the CNCF website guidelines for more details.You should read the content guide before proposing a change that adds an extra third-party link.

Windows containers in Kubernetes


Attach Handlers to Container Lifecycle EventsThis page shows how to attach handlers to Container lifecycle events. Kubernetes supports the postStart and preStop events. Kubernetes sends the postStart event immediately after a Container is started, and it sends the preStop event immediately before the Container is terminated. A Container may specify one handler per event.Before you beginYou need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. It is recommended to run this tutorial on a cluster with at least two nodes that are not acting as control plane hosts. If you do not already have a cluster, you can create one by using minikube or you can use one of these Kubernetes playgrounds:iximiuz LabsKillercodaKodeKloudPlay with KubernetesTo check the version, enter kubectl version.Define postStart and preStop handlersIn this exercise, you create a Pod that has one Container. The Container has handlers for the postStart and preStop events.Here is the configuration file for the Pod:pods/lifecycle-events.yaml apiVersion: v1 kind: Pod metadata: name: lifecycle-demo spec: containers: - name: lifecycle-demo-container image: nginx lifecycle: postStart: exec: command: ["/bin/sh", "-c", "echo Hello from the postStart handler > /usr/share/message"] preStop: exec: command: ["/bin/sh","-c","nginx -s quit; while killall -0 nginx; do sleep 1; done"] In the configuration file, you can see that the postStart command writes a message file to the Container's /usr/share directory. The preStop command shuts down nginx gracefully. This is helpful if the Container is being terminated because of a failure.Create the Pod:kubectl apply -f https://k8s.io/examples/pods/lifecycle-events.yaml Verify that the Container in the Pod is running:kubectl get pod lifecycle-demo Get a shell into the Container running in your Pod:kubectl exec -it lifecycle-demo -- /bin/bash In your shell, verify that the postStart handler created the message file:root@lifecycle-demo:/# cat /usr/share/message The output shows the text written by the postStart handler:Hello from the postStart handler DiscussionKubernetes sends the postStart event immediately after the Container is created. There is no guarantee, however, that the postStart handler is called before the Container's entrypoint is called. The postStart handler runs asynchronously relative to the Container's code, but Kubernetes' management of the container blocks until the postStart handler completes. The Container's status is not set to RUNNING until the postStart handler completes.Kubernetes sends the preStop event immediately before the Container is terminated. Kubernetes' management of the Container blocks until the preStop handler completes, unless the Pod's grace period expires. For more details, see Pod Lifecycle.Note:Kubernetes only sends the preStop event when a Pod or a container in the Pod is terminated. This means that the preStop hook is not invoked when the Pod is completed. About this limitation, please see Container hooks for the detail.What's nextLearn more about Container lifecycle hooks.Learn more about the lifecycle of a Pod.ReferenceLifecycleContainerSee terminationGracePeriodSeconds in PodSpec


Performing a Rolling UpdateObjectivesPerform a rolling update using kubectl.Updating an applicationRolling updates allow Deployments' update to take place with zero downtime by incrementally updating Pods instances with new ones.Users expect applications to be available all the time, and developers are expected to deploy new versions of them several times a day. In Kubernetes this is done with rolling updates. A rolling update allows a Deployment update to take place with zero downtime. It does this by incrementally replacing the current Pods with new ones. The new Pods are scheduled on Nodes with available resources, and Kubernetes waits for those new Pods to start before removing the old Pods.In the previous module we scaled our application to run multiple instances. This is a requirement for performing updates without affecting application availability. By default, the maximum number of Pods that can be unavailable during the update and the maximum number of new Pods that can be created, is one. Both options can be configured to either numbers or percentages (of Pods). In Kubernetes, updates are versioned and any Deployment update can be reverted to a previous (stable) version.Rolling updates overviewIf a Deployment is exposed publicly, the Service will load-balance the traffic only to available Pods during the update.Similar to application Scaling, if a Deployment is exposed publicly, the Service will load-balance the traffic only to available Pods during the update. An available Pod is an instance that is available to the users of the application.Rolling updates allow the following actions:Promote an application from one environment to another (via container image updates)Rollback to previous versionsContinuous Integration and Continuous Delivery of applications with zero downtimeIn the following interactive tutorial, we'll update our application to a new version, and also perform a rollback.Update the version of the appTo list your Deployments, run the get deployments subcommand:kubectl get deployments To list the running Pods, run the get pods subcommand:kubectl get pods To view the current image version of the app, run the describe pods subcommand and look for the Image field:kubectl describe pods To update the image of the application to version 2, use the set image subcommand, followed by the deployment name and the new image version:kubectl set image deployments/kubernetes-bootcamp kubernetes-bootcamp=docker.io/jocatalin/kubernetes-bootcamp:v2 The command notified the Deployment to use a different image for your app and initiated a rolling update. Check the status of the new Pods, and view the old one terminating with the get pods subcommand:kubectl get pods Verify an updateFirst, check that the service is running, as you might have deleted it in previous tutorial step, run describe services/kubernetes-bootcamp. If it's missing, you can create it again with:kubectl expose deployment/kubernetes-bootcamp --type="NodePort" --port 8080 Create an environment variable called NODE_PORT that has the value of the Node port assigned:export NODE_PORT="$(kubectl get services/kubernetes-bootcamp -o go-template='{{(index .spec.ports 0).nodePort}}')" echo "NODE_PORT=$NODE_PORT"

Performing a Rolling Update


Safely Drain a NodeThis page shows how to safely drain a node, optionally respecting the PodDisruptionBudget you have defined.Before you beginThis task assumes that you have met the following prerequisites:You do not require your applications to be highly available during the node drain, orYou have read about the PodDisruptionBudget concept, and have configured PodDisruptionBudgets for applications that need them.(Optional) Configure a disruption budgetTo ensure that your workloads remain available during maintenance, you can configure a PodDisruptionBudget.If availability is important for any applications that run or could run on the node(s) that you are draining, configure a PodDisruptionBudgets first and then continue following this guide.It is recommended to set AlwaysAllow Unhealthy Pod Eviction Policy to your PodDisruptionBudgets to support eviction of misbehaving applications during a node drain. The default behavior is to wait for the application pods to become healthy before the drain can proceed.Use kubectl drain to remove a node from serviceYou can use kubectl drain to safely evict all of your pods from a node before you perform maintenance on the node (e.g. kernel upgrade, hardware maintenance, etc.). Safe evictions allow the pod's containers to gracefully terminate and will respect the PodDisruptionBudgets you have specified.Note:By default kubectl drain ignores certain system pods on the node that cannot be killed; see the kubectl drain documentation for more details.When kubectl drain returns successfully, that indicates that all of the pods (except the ones excluded as described in the previous paragraph) have been safely evicted (respecting the desired graceful termination period, and respecting the PodDisruptionBudget you have defined). It is then safe to bring down the node by powering down its physical machine or, if running on a cloud platform, deleting its virtual machine.Note:If any new Pods tolerate the node.kubernetes.io/unschedulable taint, then those Pods might be scheduled to the node you have drained. Avoid tolerating that taint other than for DaemonSets.If you or another API user directly set the nodeName field for a Pod (bypassing the scheduler), then the Pod is bound to the specified node and will run there, even though you have drained that node and marked it unschedulable.First, identify the name of the node you wish to drain. You can list all of the nodes in your cluster withkubectl get nodes Next, tell Kubernetes to drain the node:kubectl drain --ignore-daemonsets <node name> If there are pods managed by a DaemonSet, you will need to specify --ignore-daemonsets with kubectl to successfully drain the node. The kubectl drain subcommand on its own does not actually drain a node of its DaemonSet pods: the DaemonSet controller (part of the control plane) immediately replaces missing Pods with new equivalent Pods. The DaemonSet controller also creates Pods that ignore unschedulable taints, which allows the new Pods to launch onto a node that you are draining.Once it returns (without giving an error), you can power down the node (or equivalently, if on a cloud platform, delete the virtual machine backing the node). If you leave the node in the cluster during the maintenance operation, you need to runkubectl uncordon <node name> afterwards to tell Kubernetes that it can resume scheduling new pods onto the node.Draining multiple nodes in parallelThe kubectl drain command should only be issued to a single node at a time. However, you can run multiple kubectl drain commands for different nodes in parallel, in different terminals or in the background. Multiple drain commands running concurrently will still respect the PodDisruptionBudget you specify.For example, if you have a StatefulSet with three replicas and have set a PodDisruptionBudget for that set specifying minAvailable: 2, kubectl drain only evicts a pod from the StatefulSet if all three replicas pods are healthy; if then you issue multiple drain commands in parallel, Kubernetes respects the PodDisruptionBudget and ensures that only 1 (calculated as replicas - minAvailable) Pod is unavailable at any given time. Any drains that would cause the number of healthy replicas to fall below the specified budget are blocked.The Eviction APIIf you prefer not to use kubectl drain (such as to avoid calling to an external command, or to get finer control over the pod eviction process), you can also programmatically cause evictions using the eviction API.For more information, see API-initiated eviction.What's nextFollow steps to protect your application by configuring a Pod Disruption Budget.

Safely Drain a Node


Understand Pressure Stall Information (PSI) MetricsDetailed explanation of Pressure Stall Information (PSI) metrics and how to use them to identify resource pressure in Kubernetes.FEATURE STATE: Kubernetes v1.34 [beta]As a beta feature, Kubernetes lets you configure the kubelet to collect Linux kernel Pressure Stall Information (PSI) for CPU, memory, and I/O usage. The information is collected at node, pod and container level. This feature is enabled by default by setting the KubeletPSI feature gate.PSI metrics are exposed through two different sources:The kubelet's Summary API, which provides PSI data at the node, pod, and container level.The /metrics/cadvisor endpoint on the kubelet, which exposes PSI metrics in the Prometheus format.RequirementsPressure Stall Information requires the following on your Linux nodes:The Linux kernel must be version 4.20 or newer.The kernel must be compiled with the CONFIG_PSI=y option. Most modern distributions enable this by default. You can check your kernel's configuration by running zgrep CONFIG_PSI /proc/config.gz.Some Linux distributions may compile PSI into the kernel but disable it by default. If so, you need to enable it at boot time by adding the psi=1 parameter to the kernel command line.The node must be using cgroup v2.Understanding PSI MetricsPressure Stall Information (PSI) metrics are provided for three resources: CPU, memory, and I/O. They are categorized into two main types of pressure: some and full.some: This value indicates that some tasks (one or more) are stalled on a resource. For example, if some tasks are waiting for I/O, this metric will increase. This can be an early indicator of resource contention.full: This value indicates that all non-idle tasks are stalled on a resource simultaneously. This signifies a more severe resource shortage, where the entire system is unable to make progress.Each pressure type provides four metrics: avg10, avg60, avg300, and total. The avg values represent the percentage of wall-clock time that tasks were stalled over 10-second, 60-second, and 3-minute moving averages. The total value is a cumulative counter in microseconds showing the total time tasks have been stalled.Example ScenariosYou can use a simple Pod with a stress-testing tool to generate resource pressure and observe the PSI metrics. The following examples use the agnhost container image, which includes the stress tool.Generating CPU PressureCreate a Pod that generates CPU pressure using the stress utility. This workload will put a heavy load on one CPU core.Create a file named cpu-pressure-pod.yaml:apiVersion: v1 kind: Pod metadata: name: cpu-pressure-pod spec: restartPolicy: Never containers: - name: cpu-stress image: registry.k8s.io/e2e-test-images/agnhost:2.47 args: - "stress" - "--cpus" - "1" resources: limits: cpu: "500m" requests: cpu: "500m" Apply it to your cluster: kubectl apply -f cpu-pressure-pod.yamlObserving CPU PressureAfter the Pod is running, you can observe the CPU pressure through either the Summary API or the Prometheus metrics endpoint.Using the Summary API:Watch the summary stats for your node. In a separate terminal, run:# Replace <node-name> with the name of a node in your cluster kubectl get --raw "/api/v1/nodes/<node-name>/proxy/stats/summary" | jq '.pods[] | select(.podRef.name | contains("cpu-pressure-pod"))' You will see the some PSI metrics for CPU increase in the summary API output. The avg10 value for some pressure should rise above zero, indicating that tasks are spending time stalled on the CPU.Using the Prometheus metrics endpoint:Query the /metrics/cadvisor endpoint to see the container_pressure_cpu_waiting_seconds_total metric.# Replace <node-name> with the name of the node where the pod is running kubectl get --raw "/api/v1/nodes/<node-name>/proxy/metrics/cadvisor" | \ grep 'container_pressure_cpu_waiting_seconds_total{container="cpu-stress"' The output should show an increasing value, indicating that the container is spending time stalled waiting for CPU resources.CleanupClean up the Pod when you are finished:kubectl delete pod cpu-pressure-pod Generating Memory PressureThis example creates a Pod that continuously writes to files in the container's writable layer, causing the kernel's page cache to grow and forcing memory reclamation, which generates pressure.Create a file named memory-pressure-pod.yaml:apiVersion: v1 kind: Pod metadata: name: memory-pressure-pod spec: restartPolicy: Never containers: - name: memory-stress image: registry.k8s.io/e2e-test-images/agnhost:2.47 command: ["/bin/sh", "-c"] args: - "i=0; while true; do dd if=/dev/zero of=testfile.$i bs=1M count=50 &>/dev/null; i=$(((i+1)%5)); sleep 0.1; done" resources: limits: memory: "200M" requests: memory: "200M" Apply it to your cluster: kubectl apply -f memory-pressure-pod.yamlObserving Memory PressureUsing the Summary API:In the summary output, you will observe an increase in the full PSI metrics for memory, indicating that the system is under significant memory pressure.# Replace <node-name> with the name of a node in your cluster kubectl get --raw "/api/v1/nodes/<node-name>/proxy/stats/summary" | jq '.pods[] | select(.podRef.name | contains("memory-pressure-pod"))' Using the Prometheus metrics endpoint:Query the /metrics/cadvisor endpoint to see the container_pressure_memory_waiting_seconds_total metric.# Replace <node-name> with the name of the node where the pod is running kubectl get --raw "/api/v1/nodes/<node-name>/proxy/metrics/cadvisor" | \ grep 'container_pressure_memory_waiting_seconds_total{container="memory-stress"' In the output, you will observe an increasing value for the metric, indicating that the system is under significant memory pressure.CleanupClean up the Pod when you are finished:kubectl delete pod memory-pressure-pod Generating I/O PressureThis Pod generates I/O pressure by repeatedly writing a file to disk and using sync to flush the data from memory, which creates I/O stalls.Create a file named io-pressure-pod.yaml:apiVersion: v1 kind: Pod metadata: name: io-pressure-pod spec: restartPolicy: Never containers: - name: io-stress image: registry.k8s.io/e2e-test-images/agnhost:2.47 command: ["/bin/sh", "-c"] args: - "while true; do dd if=/dev/zero of=testfile bs=1M count=128 &>/dev/null; sync; rm testfile &>/dev/null; done" Apply this to your cluster: kubectl apply -f io-pressure-pod.yamlObserving I/O PressureUsing the Summary API:You will see the some PSI metrics for I/O increase as the Pod continuously writes to disk.# Replace <node-name> with the name of a node in your cluster kubectl get --raw "/api/v1/nodes/<node-name>/proxy/stats/summary" | jq '.pods[] | select(.podRef.name | contains("io-pressure-pod"))' Using the Prometheus metrics endpoint:Query the /metrics/cadvisor endpoint to see the container_pressure_io_waiting_seconds_total metric.# Replace <node-name> with the name of the node where the pod is running kubectl get --raw "/api/v1/nodes/<node-name>/proxy/metrics/cadvisor" | \ grep 'container_pressure_io_waiting_seconds_total{container="io-stress"' You will see the metric's value increase as the Pod continuously writes to disk.CleanupClean up the Pod when you are finished:kubectl delete pod io-pressure-pod What's nextThe task pages for Troubleshooting Clusters discuss how to use a metrics pipeline that rely on these data.