# kube-scheduler Configuration (v1)

kube-scheduler Configuration (v1)Resource TypesDefaultPreemptionArgsInterPodAffinityArgsKubeSchedulerConfigurationNodeAffinityArgsNodeResourcesBalancedAllocationArgsNodeResourcesFitArgsPodTopologySpreadArgsVolumeBindingArgsClientConnectionConfigurationAppears in:KubeSchedulerConfigurationClientConnectionConfiguration contains details for constructing a client.FieldDescriptionkubeconfig [Required]stringkubeconfig is the path to a KubeConfig file.acceptContentTypes [Required]stringacceptContentTypes defines the Accept header sent by clients when connecting to a server, overriding the default value of 'application/json'. This field will control all connections to the server used by a particular client.contentType [Required]stringcontentType is the content type used when sending data to the server from this client.qps [Required]float32qps controls the number of queries per second allowed for this connection.burst [Required]int32burst allows extra queries to accumulate when a client is exceeding its rate.DebuggingConfigurationAppears in:KubeSchedulerConfigurationDebuggingConfiguration holds configuration for Debugging related features.FieldDescriptionenableProfiling [Required]boolenableProfiling enables profiling via web interface host:port/debug/pprof/enableContentionProfiling [Required]boolenableContentionProfiling enables block profiling, if enableProfiling is true.LeaderElectionConfigurationAppears in:KubeSchedulerConfigurationLeaderElectionConfiguration defines the configuration of leader election clients for components that can run with leader election enabled.FieldDescriptionleaderElect [Required]boolleaderElect enables a leader election client to gain leadership before executing the main loop. Enable this when running replicated components for high availability.leaseDuration [Required]meta/v1.DurationleaseDuration is the duration that non-leader candidates will wait after observing a leadership renewal until attempting to acquire leadership of a led but unrenewed leader slot. This is effectively the maximum duration that a leader can be stopped before it is replaced by another candidate. This is only applicable if leader election is enabled.renewDeadline [Required]meta/v1.DurationrenewDeadline is the interval between attempts by the acting master to renew a leadership slot before it stops leading. This must be less than or equal to the lease duration. This is only applicable if leader election is enabled.retryPeriod [Required]meta/v1.DurationretryPeriod is the duration the clients should wait between attempting acquisition and renewal of a leadership. This is only applicable if leader election is enabled.resourceLock [Required]stringresourceLock indicates the resource object type that will be used to lock during leader election cycles.resourceName [Required]stringresourceName indicates the name of resource object that will be used to lock during leader election cycles.resourceNamespace [Required]stringresourceName indicates the namespace of resource object that will be used to lock during leader election cycles.DefaultPreemptionArgsDefaultPreemptionArgs holds arguments used to configure the DefaultPreemption plugin.FieldDescriptionapiVersionstringkubescheduler.config.k8s.io/v1kindstringDefaultPreemptionArgsminCandidateNodesPercentage [Required]int32MinCandidateNodesPercentage is the minimum number of candidates to shortlist when dry running preemption as a percentage of number of nodes. Must be in the range [0, 100]. Defaults to 10% of the cluster size if unspecified.minCandidateNodesAbsolute [Required]int32MinCandidateNodesAbsolute is the absolute minimum number of candidates to shortlist. The likely number of candidates enumerated for dry running preemption is given by the formula: numCandidates = max(numNodes * minCandidateNodesPercentage, minCandidateNodesAbsolute) We say "likely" because there are other factors such as PDB violations that play a role in the number of candidates shortlisted. Must be at least 0 nodes. Defaults to 100 nodes if unspecified.InterPodAffinityArgsInterPodAffinityArgs holds arguments used to configure the InterPodAffinity plugin.FieldDescriptionapiVersionstringkubescheduler.config.k8s.io/v1kindstringInterPodAffinityArgshardPodAffinityWeight [Required]int32HardPodAffinityWeight is the scoring weight for existing pods with a matching hard affinity to the incoming pod.ignorePreferredTermsOfExistingPods [Required]boolIgnorePreferredTermsOfExistingPods configures the scheduler to ignore existing pods' preferred affinity rules when scoring candidate nodes, unless the incoming pod has inter-pod affinities.KubeSchedulerConfigurationKubeSchedulerConfiguration configures a schedulerFieldDescriptionapiVersionstringkubescheduler.config.k8s.io/v1kindstringKubeSchedulerConfigurationparallelism [Required]int32Parallelism defines the amount of parallelism in algorithms for scheduling a Pods. Must be greater than 0. Defaults to 16leaderElection [Required]LeaderElectionConfigurationLeaderElection defines the configuration of leader election client.clientConnection [Required]ClientConnectionConfigurationClientConnection specifies the kubeconfig file and client connection settings for the proxy server to use when communicating with the apiserver.DebuggingConfiguration [Required]DebuggingConfiguration(Members of DebuggingConfiguration are embedded into this type.)DebuggingConfiguration holds configuration for Debugging related features TODO: We might wanna make this a substruct like Debugging componentbaseconfigv1alpha1.DebuggingConfigurationpercentageOfNodesToScore [Required]int32PercentageOfNodesToScore is the percentage of all nodes that once found feasible for running a pod, the scheduler stops its search for more feasible nodes in the cluster. This helps improve scheduler's performance. Scheduler always tries to find at least "minFeasibleNodesToFind" feasible nodes no matter what the value of this flag is. Example: if the cluster size is 500 nodes and the value of this flag is 30, then scheduler stops finding further feasible nodes once it finds 150 feasible ones. When the value is 0, default percentage (5%--50% based on the size of the cluster) of the nodes will be scored. It is overridden by profile level PercentageOfNodesToScore.podInitialBackoffSeconds [Required]int64PodInitialBackoffSeconds is the initial backoff for unschedulable pods. If specified, it must be greater than 0. If this value is null, the default value (1s) will be used.podMaxBackoffSeconds [Required]int64PodMaxBackoffSeconds is the max backoff for unschedulable pods. If specified, it must be greater than podInitialBackoffSeconds. If this value is null, the default value (10s) will be used.profiles [Required][]KubeSchedulerProfileProfiles are scheduling profiles that kube-scheduler supports. Pods can choose to be scheduled under a particular profile by setting its associated scheduler name. Pods that don't specify any scheduler name are scheduled with the "default-scheduler" profile, if present here.extenders [Required][]ExtenderExtenders are the list of scheduler extenders, each holding the values of how to communicate with the extender. These extenders are shared by all scheduler profiles.delayCacheUntilActive [Required]boolDelayCacheUntilActive specifies when to start caching. If this is true and leader election is enabled, the scheduler will wait to fill informer caches until it is the leader. Doing so will have slower failover with the benefit of lower memory overhead while waiting to become leader. Defaults to false.NodeAffinityArgsNodeAffinityArgs holds arguments to configure the NodeAffinity plugin.FieldDescriptionapiVersionstringkubescheduler.config.k8s.io/v1kindstringNodeAffinityArgsaddedAffinitycore/v1.NodeAffinityAddedAffinity is applied to all Pods additionally to the NodeAffinity specified in the PodSpec. That is, Nodes need to satisfy AddedAffinity AND .spec.NodeAffinity. AddedAffinity is empty by default (all Nodes match). When AddedAffinity is used, some Pods with affinity requirements that match a specific Node (such as Daemonset Pods) might remain unschedulable.NodeResourcesBalancedAllocationArgsNodeResourcesBalancedAllocationArgs holds arguments used to configure NodeResourcesBalancedAllocation plugin.FieldDescriptionapiVersionstringkubescheduler.config.k8s.io/v1kindstringNodeResourcesBalancedAllocationArgsresources [Required][]ResourceSpecResources to be managed, the default is "cpu" and "memory" if not specified.NodeResourcesFitArgsNodeResourcesFitArgs holds arguments used to configure the NodeResourcesFit plugin.FieldDescriptionapiVersionstringkubescheduler.config.k8s.io/v1kindstringNodeResourcesFitArgsignoredResources [Required][]stringIgnoredResources is the list of resources that NodeResources fit filter should ignore. This doesn't apply to scoring.ignoredResourceGroups [Required][]stringIgnoredResourceGroups defines the list of resource groups that NodeResources fit filter should ignore. e.g. if group is ["example.com"], it will ignore all resource names that begin with "example.com", such as "example.com/aaa" and "example.com/bbb". A resource group name can't contain '/'. This doesn't apply to scoring.scoringStrategy [Required]ScoringStrategyScoringStrategy selects the node resource scoring strategy. The default strategy is LeastAllocated with an equal "cpu" and "memory" weight.PodTopologySpreadArgsPodTopologySpreadArgs holds arguments used to configure the PodTopologySpread plugin.FieldDescriptionapiVersionstringkubescheduler.config.k8s.io/v1kindstringPodTopologySpreadArgsdefaultConstraints[]core/v1.TopologySpreadConstraintDefaultConstraints defines topology spread constraints to be applied to Pods that don't define any in pod.spec.topologySpreadConstraints. .defaultConstraints[*].labelSelectors must be empty, as they are deduced from the Pod's membership to Services, ReplicationControllers, ReplicaSets or StatefulSets. When not empty, .defaultingType must be "List".defaultingTypePodTopologySpreadConstraintsDefaultingDefaultingType determines how .defaultConstraints are deduced. Can be one of "System" or "List"."System": Use kubernetes defined constraints that spread Pods among Nodes and Zones."List": Use constraints defined in .defaultConstraints.Defaults to "System".VolumeBindingArgsVolumeBindingArgs holds arguments used to configure the VolumeBinding plugin.FieldDescriptionapiVersionstringkubescheduler.config.k8s.io/v1kindstringVolumeBindingArgsbindTimeoutSeconds [Required]int64BindTimeoutSeconds is the timeout in seconds in volume binding operation. Value must be non-negative integer. The value zero indicates no waiting. If this value is nil, the default value (600) will be used.shape[]UtilizationShapePointShape specifies the points defining the score function shape, which is used to score nodes based on the utilization of provisioned PVs. The utilization is calculated by dividing the total requested storage of the pod by the total capacity of feasible PVs on each node. Each point contains utilization (ranges from 0 to 100) and its associated score (ranges from 0 to 10). You can turn the priority by specifying different scores for different utilization numbers. The default shape points are:10 for 0 utilization0 for 100 utilization All points must be sorted in increasing order by utilization.ExtenderAppears in:KubeSchedulerConfigurationExtender holds the parameters used to communicate with the extender. If a verb is unspecified/empty, it is assumed that the extender chose not to provide that extension.FieldDescriptionurlPrefix [Required]stringURLPrefix at which the extender is availablefilterVerb [Required]stringVerb for the filter call, empty if not supported. This verb is appended to the URLPrefix when issuing the filter call to extender.preemptVerb [Required]stringVerb for the preempt call, empty if not supported. This verb is appended to the URLPrefix when issuing the preempt call to extender.prioritizeVerb [Required]stringVerb for the prioritize call, empty if not supported. This verb is appended to the URLPrefix when issuing the prioritize call to extender.weight [Required]int64The numeric multiplier for the node scores that the prioritize call generates. The weight should be a positive integerbindVerb [Required]stringVerb for the bind call, empty if not supported. This verb is appended to the URLPrefix when issuing the bind call to extender. If this method is implemented by the extender, it is the extender's responsibility to bind the pod to apiserver. Only one extender can implement this function.enableHTTPS [Required]boolEnableHTTPS specifies whether https should be used to communicate with the extendertlsConfig [Required]ExtenderTLSConfigTLSConfig specifies the transport layer security confighttpTimeout [Required]meta/v1.DurationHTTPTimeout specifies the timeout duration for a call to the extender. Filter timeout fails the scheduling of the pod. Prioritize timeout is ignored, k8s/other extenders priorities are used to select the node.nodeCacheCapable [Required]boolNodeCacheCapable specifies that the extender is capable of caching node information, so the scheduler should only send minimal information about the eligible nodes assuming that the extender already cached full details of all nodes in the clustermanagedResources[]ExtenderManagedResourceManagedResources is a list of extended resources that are managed by this extender.A pod will be sent to the extender on the Filter, Prioritize and Bind (if the extender is the binder) phases iff the pod requests at least one of the extended resources in this list. If empty or unspecified, all pods will be sent to this extender.If IgnoredByScheduler is set to true for a resource, kube-scheduler will skip checking the resource in predicates.ignorable [Required]boolIgnorable specifies if the extender is ignorable, i.e. scheduling should not fail when the extender returns an error or is not reachable.ExtenderManagedResourceAppears in:ExtenderExtenderManagedResource describes the arguments of extended resources managed by an extender.FieldDescriptionname [Required]stringName is the extended resource name.ignoredByScheduler [Required]boolIgnoredByScheduler indicates whether kube-scheduler should ignore this resource when applying predicates.ExtenderTLSConfigAppears in:ExtenderExtenderTLSConfig contains settings to enable TLS with extenderFieldDescriptioninsecure [Required]boolServer should be accessed without verifying the TLS certificate. For testing only.serverName [Required]stringServerName is passed to the server for SNI and is used in the client to check server certificates against. If ServerName is empty, the hostname used to contact the server is used.certFile [Required]stringServer requires TLS client certificate authenticationkeyFile [Required]stringServer requires TLS client certificate authenticationcaFile [Required]stringTrusted root certificates for servercertData [Required][]byteCertData holds PEM-encoded bytes (typically read from a client certificate file). CertData takes precedence over CertFilekeyData [Required][]byteKeyData holds PEM-encoded bytes (typically read from a client certificate key file). KeyData takes precedence over KeyFilecaData [Required][]byteCAData holds PEM-encoded bytes (typically read from a root certificates bundle). CAData takes precedence over CAFileKubeSchedulerProfileAppears in:KubeSchedulerConfigurationKubeSchedulerProfile is a scheduling profile.FieldDescriptionschedulerName [Required]stringSchedulerName is the name of the scheduler associated to this profile. If SchedulerName matches with the pod's "spec.schedulerName", then the pod is scheduled with this profile.percentageOfNodesToScore [Required]int32PercentageOfNodesToScore is the percentage of all nodes that once found feasible for running a pod, the scheduler stops its search for more feasible nodes in the cluster. This helps improve scheduler's performance. Scheduler always tries to find at least "minFeasibleNodesToFind" feasible nodes no matter what the value of this flag is. Example: if the cluster size is 500 nodes and the value of this flag is 30, then scheduler stops finding further feasible nodes once it finds 150 feasible ones. When the value is 0, default percentage (5%--50% based on the size of the cluster) of the nodes will be scored. It will override global PercentageOfNodesToScore. If it is empty, global PercentageOfNodesToScore will be used.plugins [Required]PluginsPlugins specify the set of plugins that should be enabled or disabled. Enabled plugins are the ones that should be enabled in addition to the default plugins. Disabled plugins are any of the default plugins that should be disabled. When no enabled or disabled plugin is specified for an extension point, default plugins for that extension point will be used if there is any. If a QueueSort plugin is specified, the same QueueSort Plugin and PluginConfig must be specified for all profiles.pluginConfig [Required][]PluginConfigPluginConfig is an optional set of custom plugin arguments for each plugin. Omitting config args for a plugin is equivalent to using the default config for that plugin.PluginAppears in:PluginSetPlugin specifies a plugin name and its weight when applicable. Weight is used only for Score plugins.FieldDescriptionname [Required]stringName defines the name of pluginweight [Required]int32Weight defines the weight of plugin, only used for Score plugins.PluginConfigAppears in:KubeSchedulerProfilePluginConfig specifies arguments that should be passed to a plugin at the time of initialization. A plugin that is invoked at multiple extension points is initialized once. Args can have arbitrary structure. It is up to the plugin to process these Args.FieldDescriptionname [Required]stringName defines the name of plugin being configuredargs [Required]k8s.io/apimachinery/pkg/runtime.RawExtensionArgs defines the arguments passed to the plugins at the time of initialization. Args can have arbitrary structure.PluginSetAppears in:PluginsPluginSet specifies enabled and disabled plugins for an extension point. If an array is empty, missing, or nil, default plugins at that extension point will be used.FieldDescriptionenabled [Required][]PluginEnabled specifies plugins that should be enabled in addition to default plugins. If the default plugin is also configured in the scheduler config file, the weight of plugin will be overridden accordingly. These are called after default plugins and in the same order specified here.disabled [Required][]PluginDisabled specifies default plugins that should be disabled. When all default plugins need to be disabled, an array containing only one "*" should be provided.PluginsAppears in:KubeSchedulerProfilePlugins include multiple extension points. When specified, the list of plugins for a particular extension point are the only ones enabled. If an extension point is omitted from the config, then the default set of plugins is used for that extension point. Enabled plugins are called in the order specified here, after default plugins. If they need to be invoked before default plugins, default plugins must be disabled and re-enabled here in desired order.FieldDescriptionpreEnqueue [Required]PluginSetPreEnqueue is a list of plugins that should be invoked before adding pods to the scheduling queue.queueSort [Required]PluginSetQueueSort is a list of plugins that should be invoked when sorting pods in the scheduling queue.preFilter [Required]PluginSetPreFilter is a list of plugins that should be invoked at "PreFilter" extension point of the scheduling framework.filter [Required]PluginSetFilter is a list of plugins that should be invoked when filtering out nodes that cannot run the Pod.postFilter [Required]PluginSetPostFilter is a list of plugins that are invoked after filtering phase, but only when no feasible nodes were found for the pod.preScore [Required]PluginSetPreScore is a list of plugins that are invoked before scoring.score [Required]PluginSetScore is a list of plugins that should be invoked when ranking nodes that have passed the filtering phase.reserve [Required]PluginSetReserve is a list of plugins invoked when reserving/unreserving resources after a node is assigned to run the pod.permit [Required]PluginSetPermit is a list of plugins that control binding of a Pod. These plugins can prevent or delay binding of a Pod.preBind [Required]PluginSetPreBind is a list of plugins that should be invoked before a pod is bound.bind [Required]PluginSetBind is a list of plugins that should be invoked at "Bind" extension point of the scheduling framework. The scheduler call these plugins in order. Scheduler skips the rest of these plugins as soon as one returns success.postBind [Required]PluginSetPostBind is a list of plugins that should be invoked after a pod is successfully bound.multiPoint [Required]PluginSetMultiPoint is a simplified config section to enable plugins for all valid extension points. Plugins enabled through MultiPoint will automatically register for every individual extension point the plugin has implemented. Disabling a plugin through MultiPoint disables that behavior. The same is true for disabling "*" through MultiPoint (no default plugins will be automatically registered). Plugins can still be disabled through their individual extension points.In terms of precedence, plugin config follows this basic hierarchySpecific extension pointsExplicitly configured MultiPoint pluginsThe set of default plugins, as MultiPoint plugins This implies that a higher precedence plugin will run first and overwrite any settings within MultiPoint. Explicitly user-configured plugins also take a higher precedence over default plugins. Within this hierarchy, an Enabled setting takes precedence over Disabled. For example, if a plugin is set in both multiPoint.Enabled and multiPoint.Disabled, the plugin will be enabled. Similarly, including multiPoint.Disabled = '*' and multiPoint.Enabled = pluginA will still register that specific plugin through MultiPoint. This follows the same behavior as all other extension point configurations.PodTopologySpreadConstraintsDefaulting(Alias of string)Appears in:PodTopologySpreadArgsPodTopologySpreadConstraintsDefaulting defines how to set default constraints for the PodTopologySpread plugin.RequestedToCapacityRatioParamAppears in:ScoringStrategyRequestedToCapacityRatioParam define RequestedToCapacityRatio parametersFieldDescriptionshape [Required][]UtilizationShapePointShape is a list of points defining the scoring function shape.ResourceSpecAppears in:NodeResourcesBalancedAllocationArgsScoringStrategyResourceSpec represents a single resource.FieldDescriptionname [Required]stringName of the resource.weight [Required]int64Weight of the resource.ScoringStrategyAppears in:NodeResourcesFitArgsScoringStrategy define ScoringStrategyType for node resource pluginFieldDescriptiontype [Required]ScoringStrategyTypeType selects which strategy to run.resources [Required][]ResourceSpecResources to consider when scoring. The default resource set includes "cpu" and "memory" with an equal weight. Allowed weights go from 1 to 100. Weight defaults to 1 if not specified or explicitly set to 0.requestedToCapacityRatio [Required]RequestedToCapacityRatioParamArguments specific to RequestedToCapacityRatio strategy.ScoringStrategyType(Alias of string)Appears in:ScoringStrategyScoringStrategyType the type of scoring strategy used in NodeResourcesFit plugin.UtilizationShapePointAppears in:VolumeBindingArgsRequestedToCapacityRatioParamUtilizationShapePoint represents single point of priority function shape.FieldDescriptionutilization [Required]int32Utilization (x axis). Valid values are 0 to 100. Fully utilized node maps to 100.score [Required]int32Score assigned to given utilization (y axis). Valid values are 0 to 10.This page is automatically generated.If you plan to report an issue with this page, mention that the page is auto-generated in your issue description. The fix may need to happen elsewhere in the Kubernetes project.


# Configuring each kubelet in your cluster using kubeadm

Configuring each kubelet in your cluster using kubeadmNote: Dockershim has been removed from the Kubernetes project as of release 1.24. Read the Dockershim Removal FAQ for further details.FEATURE STATE: Kubernetes v1.11 [stable]The lifecycle of the kubeadm CLI tool is decoupled from the kubelet, which is a daemon that runs on each node within the Kubernetes cluster. The kubeadm CLI tool is executed by the user when Kubernetes is initialized or upgraded, whereas the kubelet is always running in the background.Since the kubelet is a daemon, it needs to be maintained by some kind of an init system or service manager. When the kubelet is installed using DEBs or RPMs, systemd is configured to manage the kubelet. You can use a different service manager instead, but you need to configure it manually.Some kubelet configuration details need to be the same across all kubelets involved in the cluster, while other configuration aspects need to be set on a per-kubelet basis to accommodate the different characteristics of a given machine (such as OS, storage, and networking). You can manage the configuration of your kubelets manually, but kubeadm now provides a KubeletConfiguration API type for managing your kubelet configurations centrally.Kubelet configuration patternsThe following sections describe patterns to kubelet configuration that are simplified by using kubeadm, rather than managing the kubelet configuration for each Node manually.Propagating cluster-level configuration to each kubeletYou can provide the kubelet with default values to be used by kubeadm init and kubeadm join commands. Interesting examples include using a different container runtime or setting the default subnet used by services.If you want your services to use the subnet 10.96.0.0/12 as the default for services, you can pass the --service-cidr parameter to kubeadm:kubeadm init --service-cidr 10.96.0.0/12 Virtual IPs for services are now allocated from this subnet. You also need to set the DNS address used by the kubelet, using the --cluster-dns flag. This setting needs to be the same for every kubelet on every manager and Node in the cluster. The kubelet provides a versioned, structured API object that can configure most parameters in the kubelet and push out this configuration to each running kubelet in the cluster. This object is called KubeletConfiguration. The KubeletConfiguration allows the user to specify flags such as the cluster DNS IP addresses expressed as a list of values to a camelCased key, illustrated by the following example:apiVersion: kubelet.config.k8s.io/v1beta1 kind: KubeletConfiguration clusterDNS: - 10.96.0.10 For more details on the KubeletConfiguration have a look at this section.Providing instance-specific configuration detailsSome hosts require specific kubelet configurations due to differences in hardware, operating system, networking, or other host-specific parameters. The following list provides a few examples.The path to the DNS resolution file, as specified by the --resolv-conf kubelet configuration flag, may differ among operating systems, or depending on whether you are using systemd-resolved. If this path is wrong, DNS resolution will fail on the Node whose kubelet is configured incorrectly.The Node API object .metadata.name is set to the machine's hostname by default, unless you are using a cloud provider. You can use the --hostname-override flag to override the default behavior if you need to specify a Node name different from the machine's hostname.Currently, the kubelet cannot automatically detect the cgroup driver used by the container runtime, but the value of --cgroup-driver must match the cgroup driver used by the container runtime to ensure the health of the kubelet.To specify the container runtime you must set its endpoint with the --container-runtime-endpoint=<path> flag.The recommended way of applying such instance-specific configuration is by using KubeletConfiguration patches.Configure kubelets using kubeadmIt is possible to configure the kubelet that kubeadm will start if a custom KubeletConfiguration API object is passed with a configuration file like so kubeadm ... --config some-config-file.yaml.By calling kubeadm config print init-defaults --component-configs KubeletConfiguration you can see all the default values for this structure.It is also possible to apply instance-specific patches over the base KubeletConfiguration. Have a look at Customizing the kubelet for more details.Workflow when using kubeadm initWhen you call kubeadm init, the kubelet configuration is marshalled to disk at /var/lib/kubelet/config.yaml, and also uploaded to a kubelet-config ConfigMap in the kube-system namespace of the cluster. Additionally, the kubeadm tool detects the CRI socket on the node and writes its details (including the socket path) into a local configuration, /var/lib/kubelet/instance-config.yaml. A kubelet configuration file is also written to /etc/kubernetes/kubelet.conf with the baseline cluster-wide configuration for all kubelets in the cluster. This configuration file points to the client certificates that allow the kubelet to communicate with the API server. This addresses the need to propagate cluster-level configuration to each kubelet.To address the second pattern of providing instance-specific configuration details, kubeadm writes an environment file to /var/lib/kubelet/kubeadm-flags.env, which contains a list of flags to pass to the kubelet when it starts. The flags are presented in the file like this:KUBELET_KUBEADM_ARGS="--flag1=value1 --flag2=value2 ..." In addition to the flags used when starting the kubelet, the file also contains dynamic parameters such as the cgroup driver.After marshalling these two files to disk, kubeadm attempts to run the following two commands, if you are using systemd:systemctl daemon-reload && systemctl restart kubelet If the reload and restart are successful, the normal kubeadm init workflow continues.Workflow when using kubeadm joinWhen you run kubeadm join, kubeadm uses the Bootstrap Token credential to perform a TLS bootstrap, which fetches the credential needed to download the kubelet-config ConfigMap and writes it to /var/lib/kubelet/config.yaml. Additionally, the kubeadm tool detects the CRI socket on the node and writes its details (including the socket path) into a local configuration, /var/lib/kubelet/instance-config.yaml. The dynamic environment file is generated in exactly the same way as kubeadm init.Next, kubeadm runs the following two commands to load the new configuration into the kubelet:systemctl daemon-reload && systemctl restart kubelet After the kubelet loads the new configuration, kubeadm writes the /etc/kubernetes/bootstrap-kubelet.conf KubeConfig file, which contains a CA certificate and Bootstrap Token. These are used by the kubelet to perform the TLS Bootstrap and obtain a unique credential, which is stored in /etc/kubernetes/kubelet.conf.When the /etc/kubernetes/kubelet.conf file is written, the kubelet has finished performing the TLS Bootstrap. Kubeadm deletes the /etc/kubernetes/bootstrap-kubelet.conf file after completing the TLS Bootstrap.The kubelet drop-in file for systemdkubeadm ships with configuration for how systemd should run the kubelet. Note that the kubeadm CLI command never touches this drop-in file.This configuration file installed by the kubeadm package is written to /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf and is used by systemd. It augments the basic kubelet.service.If you want to override that further, you can make a directory /etc/systemd/system/kubelet.service.d/ (not /usr/lib/systemd/system/kubelet.service.d/) and put your own customizations into a file there. For example, you might add a new local file /etc/systemd/system/kubelet.service.d/local-overrides.conf to override the unit settings configured by kubeadm.Here is what you are likely to find in /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf:Note:The contents below are just an example. If you don't want to use a package manager follow the guide outlined in the (Without a package manager) section.[Service] Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf" Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml" # This is a file that "kubeadm init" and "kubeadm join" generate at runtime, populating # the KUBELET_KUBEADM_ARGS variable dynamically EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env # This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably, # the user should use the .NodeRegistration.KubeletExtraArgs object in the configuration files instead. # KUBELET_EXTRA_ARGS should be sourced from this file. EnvironmentFile=-/etc/default/kubelet ExecStart= ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS This file specifies the default locations for all of the files managed by kubeadm for the kubelet.The KubeConfig file to use for the TLS Bootstrap is /etc/kubernetes/bootstrap-kubelet.conf, but it is only used if /etc/kubernetes/kubelet.conf does not exist.The KubeConfig file with the unique kubelet identity is /etc/kubernetes/kubelet.conf.The file containing the kubelet's ComponentConfig is /var/lib/kubelet/config.yaml.The dynamic environment file that contains KUBELET_KUBEADM_ARGS is sourced from /var/lib/kubelet/kubeadm-flags.env.The file that can contain user-specified flag overrides with KUBELET_EXTRA_ARGS is sourced from /etc/default/kubelet (for DEBs), or /etc/sysconfig/kubelet (for RPMs). KUBELET_EXTRA_ARGS is last in the flag chain and has the highest priority in the event of conflicting settings.Kubernetes binaries and package contentsThe DEB and RPM packages shipped with the Kubernetes releases are:Package nameDescriptionkubeadmInstalls the /usr/bin/kubeadm CLI tool and the kubelet drop-in file for the kubelet.kubeletInstalls the /usr/bin/kubelet binary.kubectlInstalls the /usr/bin/kubectl binary.cri-toolsInstalls the /usr/bin/crictl binary from the cri-tools git repository.kubernetes-cniInstalls the /opt/cni/bin binaries from the plugins git repository.


# Content organization

Content organizationThis site uses Hugo. In Hugo, content organization is a core concept.Note:Hugo Tip: Start Hugo with hugo server --navigateToChanged for content edit-sessions.Page ListsPage OrderThe documentation side menu, the documentation page browser etc. are listed using Hugo's default sort order, which sorts by weight (from 1), date (newest first), and finally by the link title.Given that, if you want to move a page or a section up, set a weight in the page's front matter:title: My Page weight: 10 Note:For page weights, it can be smart not to use 1, 2, 3 ..., but some other interval, say 10, 20, 30... This allows you to insert pages where you want later. Additionally, each weight within the same directory (section) should not be overlapped with the other weights. This makes sure that content is always organized correctly, especially in localized content.Documentation Main MenuThe Documentation main menu is built from the sections below docs/ with the main_menu flag set in front matter of the _index.md section content file:main_menu: true Note that the link title is fetched from the page's linkTitle, so if you want it to be something different than the title, change it in the content file:main_menu: true title: Page Title linkTitle: Title used in links Note:The above needs to be done per language. If you don't see your section in the menu, it is probably because it is not identified as a section by Hugo. Create a _index.md content file in the section folder.Documentation Side MenuThe documentation side-bar menu is built from the current section tree starting below docs/.It will show all sections and their pages.If you don't want to list a section or page, set the toc_hide flag to true in front matter:toc_hide: true When you navigate to a section that has content, the specific section or page (e.g. _index.md) is shown. Else, the first page inside that section is shown.Documentation BrowserThe page browser on the documentation home page is built using all the sections and pages that are directly below the docs section.If you don't want to list a section or page, set the toc_hide flag to true in front matter:toc_hide: true The Main MenuThe site links in the top-right menu -- and also in the footer -- are built by page-lookups. This is to make sure that the page actually exists. So, if the case-studies section does not exist in a site (language), it will not be linked to.Page BundlesIn addition to standalone content pages (Markdown files), Hugo supports Page Bundles.One example is Custom Hugo Shortcodes. It is considered a leaf bundle. Everything below the directory, including the index.md, will be part of the bundle. This also includes page-relative links, images that can be processed etc.:en/docs/home/contribute/includes ├── example1.md ├── example2.md ├── index.md └── podtemplate.json Another widely used example is the includes bundle. It sets headless: true in front matter, which means that it does not get its own URL. It is only used in other pages.en/includes ├── default-storage-class-prereqs.md ├── index.md ├── partner-script.js ├── partner-style.css ├── task-tutorial-prereqs.md ├── user-guide-content-moved.md └── user-guide-migration-notice.md Some important notes to the files in the bundles:For translated bundles, any missing non-content files will be inherited from languages above. This avoids duplication.All the files in a bundle are what Hugo calls Resources and you can provide metadata per language, such as parameters and title, even if it does not supports front matter (YAML files etc.). See Page Resources Metadata.The value you get from .RelPermalink of a Resource is page-relative. See Permalinks.StylesThe SASS source of the stylesheets for this site is stored in assets/sass and is automatically built by Hugo.What's nextLearn about custom Hugo shortcodesLearn about the Style guideLearn about the Content guide


# Concepts

This is the multi-page printable view of this section. Click here to print.Return to the regular view of this page.Concepts1: Overview1.1: Kubernetes Components1.2: Objects In Kubernetes1.2.1: Kubernetes Object Management1.2.2: Object Names and IDs1.2.3: Labels and Selectors1.2.4: Namespaces1.2.5: Annotations1.2.6: Field Selectors1.2.7: Finalizers1.2.8: Owners and Dependents1.2.9: Recommended Labels1.3: The Kubernetes API2: Cluster Architecture2.1: Nodes2.2: Communication between Nodes and the Control Plane2.3: Controllers2.4: Leases2.5: Cloud Controller Manager2.6: About cgroup v22.7: Kubernetes Self-Healing2.8: Container Runtime Interface (CRI)2.9: Garbage Collection2.10: Mixed Version Proxy3: Containers3.1: Images3.2: Container Environment3.3: Runtime Class3.4: Container Lifecycle Hooks4: Workloads4.1: Pods4.1.1: Pod Lifecycle4.1.2: Init Containers4.1.3: Sidecar Containers4.1.4: Ephemeral Containers4.1.5: Disruptions4.1.6: Pod Hostname4.1.7: Pod Quality of Service Classes4.1.8: User Namespaces4.1.9: Downward API4.2: Workload Management4.2.1: Deployments4.2.2: ReplicaSet4.2.3: StatefulSets4.2.4: DaemonSet4.2.5: Jobs4.2.6: Automatic Cleanup for Finished Jobs4.2.7: CronJob4.2.8: ReplicationController4.3: Autoscaling Workloads4.4: Managing Workloads5: Services, Load Balancing, and Networking5.1: Service5.2: Ingress5.3: Ingress Controllers5.4: Gateway API5.5: EndpointSlices5.6: Network Policies5.7: DNS for Services and Pods5.8: IPv4/IPv6 dual-stack5.9: Topology Aware Routing5.10: Networking on Windows5.11: Service ClusterIP allocation5.12: Service Internal Traffic Policy6: Storage6.1: Volumes6.2: Persistent Volumes6.3: Projected Volumes6.4: Ephemeral Volumes6.5: Storage Classes6.6: Volume Attributes Classes6.7: Dynamic Volume Provisioning6.8: Volume Snapshots6.9: Volume Snapshot Classes6.10: CSI Volume Cloning6.11: Storage Capacity6.12: Node-specific Volume Limits6.13: Volume Health Monitoring6.14: Windows Storage7: Configuration7.1: Configuration Best Practices7.2: ConfigMaps7.3: Secrets7.4: Liveness, Readiness, and Startup Probes7.5: Resource Management for Pods and Containers7.6: Organizing Cluster Access Using kubeconfig Files7.7: Resource Management for Windows nodes8: Security8.1: Cloud Native Security and Kubernetes8.2: Pod Security Standards8.3: Pod Security Admission8.4: Service Accounts8.5: Pod Security Policies8.6: Security For Linux Nodes8.7: Security For Windows Nodes8.8: Controlling Access to the Kubernetes API8.9: Role Based Access Control Good Practices8.10: Good practices for Kubernetes Secrets8.11: Multi-tenancy8.12: Hardening Guide - Authentication Mechanisms8.13: Hardening Guide - Scheduler Configuration8.14: Kubernetes API Server Bypass Risks8.15: Linux kernel security constraints for Pods and containers8.16: Security Checklist8.17: Application Security Checklist9: Policies9.1: Limit Ranges9.2: Resource Quotas9.3: Process ID Limits And Reservations9.4: Node Resource Managers10: Scheduling, Preemption and Eviction10.1: Kubernetes Scheduler10.2: Assigning Pods to Nodes10.3: Pod Overhead10.4: Pod Scheduling Readiness10.5: Pod Topology Spread Constraints10.6: Taints and Tolerations10.7: Scheduling Framework10.8: Dynamic Resource Allocation10.9: Scheduler Performance Tuning10.10: Resource Bin Packing10.11: Pod Priority and Preemption10.12: Node-pressure Eviction10.13: API-initiated Eviction11: Cluster Administration11.1: Node Shutdowns11.2: Swap memory management11.3: Node Autoscaling11.4: Certificates11.5: Cluster Networking11.6: Admission Webhook Good Practices11.7: Good practices for Dynamic Resource Allocation as a Cluster Admin11.8: Logging Architecture11.9: Compatibility Version For Kubernetes Control Plane Components11.10: Metrics For Kubernetes System Components11.11: Metrics for Kubernetes Object States11.12: System Logs11.13: Traces For Kubernetes System Components11.14: Proxies in Kubernetes11.15: API Priority and Fairness11.16: Installing Addons11.17: Coordinated Leader Election12: Windows in Kubernetes12.1: Windows containers in Kubernetes12.2: Guide for Running Windows Containers in Kubernetes13: Extending Kubernetes13.1: Compute, Storage, and Networking Extensions13.1.1: Network Plugins13.1.2: Device Plugins13.2: Extending the Kubernetes API13.2.1: Custom Resources13.2.2: Kubernetes API Aggregation Layer13.3: Operator patternThe Concepts section helps you learn about the parts of the Kubernetes system and the abstractions Kubernetes uses to represent your cluster, and helps you obtain a deeper understanding of how Kubernetes works.1 - OverviewKubernetes is a portable, extensible, open source platform for managing containerized workloads and services, that facilitates both declarative configuration and automation. It has a large, rapidly growing ecosystem. Kubernetes services, support, and tools are widely available.This page is an overview of Kubernetes.The name Kubernetes originates from Greek, meaning helmsman or pilot. K8s as an abbreviation results from counting the eight letters between the "K" and the "s". Google open-sourced the Kubernetes project in 2014. Kubernetes combines over 15 years of Google's experience running production workloads at scale with best-of-breed ideas and practices from the community.Why you need Kubernetes and what it can doContainers are a good way to bundle and run your applications. In a production environment, you need to manage the containers that run the applications and ensure that there is no downtime. For example, if a container goes down, another container needs to start. Wouldn't it be easier if this behavior was handled by a system?That's how Kubernetes comes to the rescue! Kubernetes provides you with a framework to run distributed systems resiliently. It takes care of scaling and failover for your application, provides deployment patterns, and more. For example: Kubernetes can easily manage a canary deployment for your system.Kubernetes provides you with:Service discovery and load balancing Kubernetes can expose a container using the DNS name or using their own IP address. If traffic to a container is high, Kubernetes is able to load balance and distribute the network traffic so that the deployment is stable.Storage orchestration Kubernetes allows you to automatically mount a storage system of your choice, such as local storages, public cloud providers, and more.Automated rollouts and rollbacks You can describe the desired state for your deployed containers using Kubernetes, and it can change the actual state to the desired state at a controlled rate. For example, you can automate Kubernetes to create new containers for your deployment, remove existing containers and adopt all their resources to the new container.Automatic bin packing You provide Kubernetes with a cluster of nodes that it can use to run containerized tasks. You tell Kubernetes how much CPU and memory (RAM) each container needs. Kubernetes can fit containers onto your nodes to make the best use of your resources.Self-healing Kubernetes restarts containers that fail, replaces containers, kills containers that don't respond to your user-defined health check, and doesn't advertise them to clients until they are ready to serve.Secret and configuration management Kubernetes lets you store and manage sensitive information, such as passwords, OAuth tokens, and SSH keys. You can deploy and update secrets and application configuration without rebuilding your container images, and without exposing secrets in your stack configuration.Batch execution In addition to services, Kubernetes can manage your batch and CI workloads, replacing containers that fail, if desired.Horizontal scaling Scale your application up and down with a simple command, with a UI, or automatically based on CPU usage.IPv4/IPv6 dual-stack Allocation of IPv4 and IPv6 addresses to Pods and ServicesDesigned for extensibility Add features to your Kubernetes cluster without changing upstream source code.What Kubernetes is notKubernetes is not a traditional, all-inclusive PaaS (Platform as a Service) system. Since Kubernetes operates at the container level rather than at the hardware level, it provides some generally applicable features common to PaaS offerings, such as deployment, scaling, load balancing, and lets users integrate their logging, monitoring, and alerting solutions. However, Kubernetes is not monolithic, and these default solutions are optional and pluggable. Kubernetes provides the building blocks for building developer platforms, but preserves user choice and flexibility where it is important.Kubernetes:Does not limit the types of applications supported. Kubernetes aims to support an extremely diverse variety of workloads, including stateless, stateful, and data-processing workloads. If an application can run in a container, it should run great on Kubernetes.Does not deploy source code and does not build your application. Continuous Integration, Delivery, and Deployment (CI/CD) workflows are determined by organization cultures and preferences as well as technical requirements.Does not provide application-level services, such as middleware (for example, message buses), data-processing frameworks (for example, Spark), databases (for example, MySQL), caches, nor cluster storage systems (for example, Ceph) as built-in services. Such components can run on Kubernetes, and/or can be accessed by applications running on Kubernetes through portable mechanisms, such as the Open Service Broker.Does not dictate logging, monitoring, or alerting solutions. It provides some integrations as proof of concept, and mechanisms to collect and export metrics.Does not provide nor mandate a configuration language/system (for example, Jsonnet). It provides a declarative API that may be targeted by arbitrary forms of declarative specifications.Does not provide nor adopt any comprehensive machine configuration, maintenance, management, or self-healing systems.Additionally, Kubernetes is not a mere orchestration system. In fact, it eliminates the need for orchestration. The technical definition of orchestration is execution of a defined workflow: first do A, then B, then C. In contrast, Kubernetes comprises a set of independent, composable control processes that continuously drive the current state towards the provided desired state. It shouldn't matter how you get from A to C. Centralized control is also not required. This results in a system that is easier to use and more powerful, robust, resilient, and extensible.Historical context for KubernetesLet's take a look at why Kubernetes is so useful by going back in time.Traditional deployment era:Early on, organizations ran applications on physical servers. There was no way to define resource boundaries for applications in a physical server, and this caused resource allocation issues. For example, if multiple applications run on a physical server, there can be instances where one application would take up most of the resources, and as a result, the other applications would underperform. A solution for this would be to run each application on a different physical server. But this did not scale as resources were underutilized, and it was expensive for organizations to maintain many physical servers.Virtualized deployment era:As a solution, virtualization was introduced. It allows you to run multiple Virtual Machines (VMs) on a single physical server's CPU. Virtualization allows applications to be isolated between VMs and provides a level of security as the information of one application cannot be freely accessed by another application.Virtualization allows better utilization of resources in a physical server and allows better scalability because an application can be added or updated easily, reduces hardware costs, and much more. With virtualization you can present a set of physical resources as a cluster of disposable virtual machines.Each VM is a full machine running all the components, including its own operating system, on top of the virtualized hardware.Container deployment era:Containers are similar to VMs, but they have relaxed isolation properties to share the Operating System (OS) among the applications. Therefore, containers are considered lightweight. Similar to a VM, a container has its own filesystem, share of CPU, memory, process space, and more. As they are decoupled from the underlying infrastructure, they are portable across clouds and OS distributions.Containers have become popular because they provide extra benefits, such as:Agile application creation and deployment: increased ease and efficiency of container image creation compared to VM image use.Continuous development, integration, and deployment: provides for reliable and frequent container image build and deployment with quick and efficient rollbacks (due to image immutability).Dev and Ops separation of concerns: create application container images at build/release time rather than deployment time, thereby decoupling applications from infrastructure.Observability: not only surfaces OS-level information and metrics, but also application health and other signals.Environmental consistency across development, testing, and production: runs the same on a laptop as it does in the cloud.Cloud and OS distribution portability: runs on Ubuntu, RHEL, CoreOS, on-premises, on major public clouds, and anywhere else.Application-centric management: raises the level of abstraction from running an OS on virtual hardware to running an application on an OS using logical resources.Loosely coupled, distributed, elastic, liberated micro-services: applications are broken into smaller, independent pieces and can be deployed and managed dynamically – not a monolithic stack running on one big single-purpose machine.Resource isolation: predictable application performance.Resource utilization: high efficiency and density.What's