# kube-apiserver

kube-apiserverSynopsisThe Kubernetes API server validates and configures data for the api objects which include pods, services, replicationcontrollers, and others. The API Server services REST operations and provides the frontend to the cluster's shared state through which all other components interact.kube-apiserver [flags] Options--admission-control-config-file stringFile with admission control configuration.--advertise-address stringThe IP address on which to advertise the apiserver to members of the cluster. This address must be reachable by the rest of the cluster. If blank, the --bind-address will be used. If --bind-address is unspecified, the host's default interface will be used.--aggregator-reject-forwarding-redirect Default: trueAggregator reject forwarding redirect response back to client.--allow-metric-labels stringToString Default: []The map from metric-label to value allow-list of this label. The key's format is <MetricName>,<LabelName>. The value's format is <allowed_value>,<allowed_value>...e.g. metric1,label1='v1,v2,v3', metric1,label2='v1,v2,v3' metric2,label1='v1,v2,v3'.--allow-metric-labels-manifest stringThe path to the manifest file that contains the allow-list mapping. The format of the file is the same as the flag --allow-metric-labels. Note that the flag --allow-metric-labels will override the manifest file.--allow-privilegedIf true, allow privileged containers. [default=false]--anonymous-auth Default: trueEnables anonymous requests to the secure port of the API server. Requests that are not rejected by another authentication method are treated as anonymous requests. Anonymous requests have a username of system:anonymous, and a group name of system:unauthenticated.--api-audiences stringsIdentifiers of the API. The service account token authenticator will validate that tokens used against the API are bound to at least one of these audiences. If the --service-account-issuer flag is configured and this flag is not, this field defaults to a single element list containing the issuer URL.--audit-log-batch-buffer-size int Default: 10000The size of the buffer to store events before batching and writing. Only used in batch mode.--audit-log-batch-max-size int Default: 1The maximum size of a batch. Only used in batch mode.--audit-log-batch-max-wait durationThe amount of time to wait before force writing the batch that hadn't reached the max size. Only used in batch mode.--audit-log-batch-throttle-burst intMaximum number of requests sent at the same moment if ThrottleQPS was not utilized before. Only used in batch mode.--audit-log-batch-throttle-enableWhether batching throttling is enabled. Only used in batch mode.--audit-log-batch-throttle-qps floatMaximum average number of batches per second. Only used in batch mode.--audit-log-compressIf set, the rotated log files will be compressed using gzip.--audit-log-format string Default: "json"Format of saved audits. "legacy" indicates 1-line text format for each event. "json" indicates structured json format. Known formats are legacy,json.--audit-log-maxage intThe maximum number of days to retain old audit log files based on the timestamp encoded in their filename.--audit-log-maxbackup intThe maximum number of old audit log files to retain. Setting a value of 0 will mean there's no restriction on the number of files.--audit-log-maxsize intThe maximum size in megabytes of the audit log file before it gets rotated.--audit-log-mode string Default: "blocking"Strategy for sending audit events. Blocking indicates sending events should block server responses. Batch causes the backend to buffer and write events asynchronously. Known modes are batch,blocking,blocking-strict.--audit-log-path stringIf set, all requests coming to the apiserver will be logged to this file. '-' means standard out.--audit-log-truncate-enabledWhether event and batch truncating is enabled.--audit-log-truncate-max-batch-size int Default: 10485760Maximum size of the batch sent to the underlying backend. Actual serialized size can be several hundreds of bytes greater. If a batch exceeds this limit, it is split into several batches of smaller size.--audit-log-truncate-max-event-size int Default: 102400Maximum size of the audit event sent to the underlying backend. If the size of an event is greater than this number, first request and response are removed, and if this doesn't reduce the size enough, event is discarded.--audit-log-version string Default: "audit.k8s.io/v1"API group and version used for serializing audit events written to log.--audit-policy-file stringPath to the file that defines the audit policy configuration.--audit-webhook-batch-buffer-size int Default: 10000The size of the buffer to store events before batching and writing. Only used in batch mode.--audit-webhook-batch-max-size int Default: 400The maximum size of a batch. Only used in batch mode.--audit-webhook-batch-max-wait duration Default: 30sThe amount of time to wait before force writing the batch that hadn't reached the max size. Only used in batch mode.--audit-webhook-batch-throttle-burst int Default: 15Maximum number of requests sent at the same moment if ThrottleQPS was not utilized before. Only used in batch mode.--audit-webhook-batch-throttle-enable Default: trueWhether batching throttling is enabled. Only used in batch mode.--audit-webhook-batch-throttle-qps float Default: 10Maximum average number of batches per second. Only used in batch mode.--audit-webhook-config-file stringPath to a kubeconfig formatted file that defines the audit webhook configuration.--audit-webhook-initial-backoff duration Default: 10sThe amount of time to wait before retrying the first failed request.--audit-webhook-mode string Default: "batch"Strategy for sending audit events. Blocking indicates sending events should block server responses. Batch causes the backend to buffer and write events asynchronously. Known modes are batch,blocking,blocking-strict.--audit-webhook-truncate-enabledWhether event and batch truncating is enabled.--audit-webhook-truncate-max-batch-size int Default: 10485760Maximum size of the batch sent to the underlying backend. Actual serialized size can be several hundreds of bytes greater. If a batch exceeds this limit, it is split into several batches of smaller size.--audit-webhook-truncate-max-event-size int Default: 102400Maximum size of the audit event sent to the underlying backend. If the size of an event is greater than this number, first request and response are removed, and if this doesn't reduce the size enough, event is discarded.--audit-webhook-version string Default: "audit.k8s.io/v1"API group and version used for serializing audit events written to webhook.--authentication-config stringFile with Authentication Configuration to configure the JWT Token authenticator or the anonymous authenticator. Requires the StructuredAuthenticationConfiguration feature gate. Also requires the feature gate AnonymousAuthConfigurableEndpoints to configure the anonymous authenticator in the config file. This flag is mutually exclusive with the --oidc-* flags if the file configures the JWT Token authenticator. This flag is mutually exclusive with --anonymous-auth if the file configures the Anonymous authenticator.--authentication-token-webhook-cache-ttl duration Default: 2m0sThe duration to cache responses from the webhook token authenticator.--authentication-token-webhook-config-file stringFile with webhook configuration for token authentication in kubeconfig format. The API server will query the remote service to determine authentication for bearer tokens.--authentication-token-webhook-version string Default: "v1beta1"The API version of the authentication.k8s.io TokenReview to send to and expect from the webhook.--authorization-config stringFile with Authorization Configuration to configure the authorizer chain. Requires feature gate StructuredAuthorizationConfiguration. This flag is mutually exclusive with the other --authorization-mode and --authorization-webhook-* flags.--authorization-mode stringsOrdered list of plug-ins to do authorization on secure port. Defaults to AlwaysAllow if --authorization-config is not used. Comma-delimited list of: AlwaysAllow,AlwaysDeny,ABAC,Webhook,RBAC,Node.--authorization-policy-file stringFile with authorization policy in json line by line format, used with --authorization-mode=ABAC, on the secure port.--authorization-webhook-cache-authorized-ttl duration Default: 5m0sThe duration to cache 'authorized' responses from the webhook authorizer.--authorization-webhook-cache-unauthorized-ttl duration Default: 30sThe duration to cache 'unauthorized' responses from the webhook authorizer.--authorization-webhook-config-file stringFile with webhook configuration in kubeconfig format, used with --authorization-mode=Webhook. The API server will query the remote service to determine access on the API server's secure port.--authorization-webhook-version string Default: "v1beta1"The API version of the authorization.k8s.io SubjectAccessReview to send to and expect from the webhook.--bind-address string Default: 0.0.0.0The IP address on which to listen for the --secure-port port. The associated interface(s) must be reachable by the rest of the cluster, and by CLI/web clients. If blank or an unspecified address (0.0.0.0 or ::), all interfaces and IP address families will be used.--cert-dir string Default: "/var/run/kubernetes"The directory where the TLS certs are located. If --tls-cert-file and --tls-private-key-file are provided, this flag will be ignored.--client-ca-file stringIf set, any request presenting a client certificate signed by one of the authorities in the client-ca-file is authenticated with an identity corresponding to the CommonName of the client certificate.--contention-profilingEnable block profiling, if profiling is enabled--cors-allowed-origins stringsList of allowed origins for CORS, comma separated. An allowed origin can be a regular expression to support subdomain matching. If this list is empty CORS will not be enabled. Please ensure each expression matches the entire hostname by anchoring to the start with '^' or including the '//' prefix, and by anchoring to the end with '$' or including the ':' port separator suffix. Examples of valid expressions are '//example.com(:|$)' and '^https://example.com(:|$)'--debug-socket-path stringUse an unprotected (no authn/authz) unix-domain socket for profiling with the given path--default-not-ready-toleration-seconds int Default: 300Indicates the tolerationSeconds of the toleration for notReady:NoExecute that is added by default to every pod that does not already have such a toleration.--default-unreachable-toleration-seconds int Default: 300Indicates the tolerationSeconds of the toleration for unreachable:NoExecute that is added by default to every pod that does not already have such a toleration.--delete-collection-workers int Default: 1Number of workers spawned for DeleteCollection call. These are used to speed up namespace cleanup.--disable-admission-plugins stringsadmission plugins that should be disabled although they are in the default enabled plugins list (NamespaceLifecycle, LimitRanger, ServiceAccount, TaintNodesByCondition, PodSecurity, Priority, DefaultTolerationSeconds, DefaultStorageClass, StorageObjectInUseProtection, PersistentVolumeClaimResize, RuntimeClass, CertificateApproval, CertificateSigning, ClusterTrustBundleAttest, CertificateSubjectRestriction, DefaultIngressClass, PodTopologyLabels, MutatingAdmissionPolicy, MutatingAdmissionWebhook, ValidatingAdmissionPolicy, ValidatingAdmissionWebhook, ResourceQuota). Comma-delimited list of admission plugins: AlwaysAdmit, AlwaysDeny, AlwaysPullImages, CertificateApproval, CertificateSigning, CertificateSubjectRestriction, ClusterTrustBundleAttest, DefaultIngressClass, DefaultStorageClass, DefaultTolerationSeconds, DenyServiceExternalIPs, EventRateLimit, ExtendedResourceToleration, ImagePolicyWebhook, LimitPodHardAntiAffinityTopology, LimitRanger, MutatingAdmissionPolicy, MutatingAdmissionWebhook, NamespaceAutoProvision, NamespaceExists, NamespaceLifecycle, NodeRestriction, OwnerReferencesPermissionEnforcement, PersistentVolumeClaimResize, PodNodeSelector, PodSecurity, PodTolerationRestriction, PodTopologyLabels, Priority, ResourceQuota, RuntimeClass, ServiceAccount, StorageObjectInUseProtection, TaintNodesByCondition, ValidatingAdmissionPolicy, ValidatingAdmissionWebhook. The order of plugins in this flag does not matter.--disable-http2-servingIf true, HTTP2 serving will be disabled [default=false]--disabled-metrics stringsThis flag provides an escape hatch for misbehaving metrics. You must provide the fully qualified metric name in order to disable it. Disclaimer: disabling metrics is higher in precedence than showing hidden metrics.--egress-selector-config-file stringFile with apiserver egress selector configuration.--emulated-version stringsThe versions different components emulate their capabilities (APIs, features, ...) of.If set, the component will emulate the behavior of this version instead of the underlying binary version.Version format could only be major.minor, for example: '--emulated-version=wardle=1.2,kube=1.31'.Options are: kube=1.31..1.33(default:1.33)If the component is not specified, defaults to "kube"--emulation-forward-compatibleIf true, for any beta+ APIs enabled by default or by --runtime-config at the emulation version, their future versions with higher priority/stability will be auto enabled even if they introduced after the emulation version. Can only be set to true if the emulation version is lower than the binary version.--enable-admission-plugins stringsadmission plugins that should be enabled in addition to default enabled ones (NamespaceLifecycle, LimitRanger, ServiceAccount, TaintNodesByCondition, PodSecurity, Priority, DefaultTolerationSeconds, DefaultStorageClass, StorageObjectInUseProtection, PersistentVolumeClaimResize, RuntimeClass, CertificateApproval, CertificateSigning, ClusterTrustBundleAttest, CertificateSubjectRestriction, DefaultIngressClass, PodTopologyLabels, MutatingAdmissionPolicy, MutatingAdmissionWebhook, ValidatingAdmissionPolicy, ValidatingAdmissionWebhook, ResourceQuota). Comma-delimited list of admission plugins: AlwaysAdmit, AlwaysDeny, AlwaysPullImages, CertificateApproval, CertificateSigning, CertificateSubjectRestriction, ClusterTrustBundleAttest, DefaultIngressClass, DefaultStorageClass, DefaultTolerationSeconds, DenyServiceExternalIPs, EventRateLimit, ExtendedResourceToleration, ImagePolicyWebhook, LimitPodHardAntiAffinityTopology, LimitRanger, MutatingAdmissionPolicy, MutatingAdmissionWebhook, NamespaceAutoProvision, NamespaceExists, NamespaceLifecycle, NodeRestriction, OwnerReferencesPermissionEnforcement, PersistentVolumeClaimResize, PodNodeSelector, PodSecurity, PodTolerationRestriction, PodTopologyLabels, Priority, ResourceQuota, RuntimeClass, ServiceAccount, StorageObjectInUseProtection, TaintNodesByCondition, ValidatingAdmissionPolicy, ValidatingAdmissionWebhook. The order of plugins in this flag does not matter.--enable-aggregator-routingTurns on aggregator routing requests to endpoints IP rather than cluster IP.--enable-bootstrap-token-authEnable to allow secrets of type 'bootstrap.kubernetes.io/token' in the 'kube-system' namespace to be used for TLS bootstrapping authentication.--enable-garbage-collector Default: trueEnables the generic garbage collector. MUST be synced with the corresponding flag of the kube-controller-manager.--enable-priority-and-fairness Default: trueIf true, replace the max-in-flight handler with an enhanced one that queues and dispatches with priority and fairness--encryption-provider-config stringThe file containing configuration for encryption providers to be used for storing secrets in etcd--encryption-provider-config-automatic-reloadDetermines if the file set by --encryption-provider-config should be automatically reloaded if the disk contents change. Setting this to true disables the ability to uniquely identify distinct KMS plugins via the API server healthz endpoints.--endpoint-reconciler-type string Default: "lease"Use an endpoint reconciler (master-count, lease, none) master-count is deprecated, and will be removed in a future version.--etcd-cafile stringSSL Certificate Authority file used to secure etcd communication.--etcd-certfile stringSSL certification file used to secure etcd communication.--etcd-compaction-interval duration Default: 5m0sThe interval of compaction requests. If 0, the compaction request from apiserver is disabled.--etcd-count-metric-poll-period duration Default: 1m0sFrequency of polling etcd for number of resources per type. 0 disables the metric collection.--etcd-db-metric-poll-interval duration Default: 30sThe interval of requests to poll etcd and update metric. 0 disables the metric collection--etcd-healthcheck-timeout duration Default: 2sThe timeout to use when checking etcd health.--etcd-keyfile stringSSL key file used to secure etcd communication.--etcd-prefix string Default: "/registry"The prefix to prepend to all resource paths in etcd.--etcd-readycheck-timeout duration Default: 2sThe timeout to use when checking etcd readiness--etcd-servers stringsList of etcd servers to connect with (scheme://ip:port), comma separated.--etcd-servers-overrides stringsPer-resource etcd servers overrides, comma separated. The individual override format: group/resource#servers, where servers are URLs, semicolon separated. Note that this applies only to resources compiled into this server binary.--event-ttl duration Default: 1h0m0sAmount of time to retain events.--external-hostname stringThe hostname to use when generating externalized URLs for this master (e.g. Swagger API Docs or OpenID Discovery).--feature-gates colonSeparatedMultimapStringStringComma-separated list of component:key=value pairs that describe feature gates for alpha/experimental features of different components.If the component is not specified, defaults to "kube". This flag can be repeatedly invoked. For example: --feature-gates 'wardle:featureA=true,wardle:featureB=false' --feature-gates 'kube:featureC=true'Options are:kube:APIResponseCompression=true|false (BETA - default=true)kube:APIServerIdentity=true|false (BETA - default=true)kube:APIServerTracing=true|false (BETA - default=true)kube:APIServingWithRoutine=true|false (ALPHA - default=false)kube:AllAlpha=true|false (ALPHA - default=false)kube:AllBeta=true|false (BETA - default=false)kube:AllowParsingUserUIDFromCertAuth=true|false (BETA - default=true)kube:AllowUnsafeMalformedObjectDeletion=true|false (ALPHA - default=false)kube:AnonymousAuthConfigurableEndpoints=true|false (BETA - default=true)kube:AuthorizeNodeWithSelectors=true|false (BETA - default=true)kube:AuthorizeWithSelectors=true|false (BETA - default=true)kube:CBORServingAndStorage=true|false (ALPHA - default=false)kube:CPUManagerPolicyAlphaOptions=true|false (ALPHA - default=false)kube:CPUManagerPolicyBetaOptions=true|false (BETA - default=true)kube:CSIVolumeHealth=true|false (ALPHA - default=false)kube:ClientsAllowCBOR=true|false (ALPHA - default=false)kube:ClientsPreferCBOR=true|false (ALPHA - default=false)kube:CloudControllerManagerWebhook=true|false (ALPHA - default=false)kube:ClusterTrustBundle=true|false (BETA - default=false)kube:ClusterTrustBundleProjection=true|false (BETA - default=false)kube:ComponentFlagz=true|false (ALPHA - default=false)kube:ComponentStatusz=true|false (ALPHA - default=false)kube:ConcurrentWatchObjectDecode=true|false (BETA - default=false)kube:ConsistentListFromCache=true|false (BETA - default=true)kube:ContainerCheckpoint=true|false (BETA - default=true)kube:ContainerStopSignals=true|false (ALPHA - default=false)kube:ContextualLogging=true|false (BETA - default=true)kube:CoordinatedLeaderElection=true|false (BETA - default=false)kube:CrossNamespaceVolumeDataSource=true|false (ALPHA - default=false)kube:CustomCPUCFSQuotaPeriod=true|false (ALPHA - default=false)kube:DRAAdminAccess=true|false (ALPHA - default=false)kube:DRADeviceTaints=true|false (ALPHA - default=false)kube:DRAPartitionableDevices=true|false (ALPHA - default=false)kube:DRAPrioritizedList=true|false (ALPHA - default=false)kube:DRAResourceClaimDeviceStatus=true|false (BETA - default=true)kube:DeclarativeValidation=true|false (BETA - default=true)kube:DeclarativeValidationTakeover=true|false (BETA - default=false)kube:DeploymentReplicaSetTerminatingReplicas=true|false (ALPHA - default=false)kube:DisableAllocatorDualWrite=true|false (BETA - default=false)kube:DisableCPUQuotaWithExclusiveCPUs=true|false (BETA - default=true)kube:DynamicResourceAllocation=true|false (BETA - default=false)kube:EventedPLEG=true|false (ALPHA - default=false)kube:ExternalServiceAccountTokenSigner=true|false (ALPHA - default=false)kube:GracefulNodeShutdown=true|false (BETA - default=true)kube:GracefulNodeShutdownBasedOnPodPriority=true|false (BETA - default=true)kube:HPAConfigurableTolerance=true|false (ALPHA - default=false)kube:HPAScaleToZero=true|false (ALPHA - default=false)kube:ImageMaximumGCAge=true|false (BETA - default=true)kube:ImageVolume=true|false (BETA - default=false)kube:InOrderInformers=true|false (BETA - default=true)kube:InPlacePodVerticalScaling=true|false (BETA - default=true)kube:InPlacePodVerticalScalingExclusiveCPUs=true|false (ALPHA - default=false)kube:InTreePluginPortworxUnregister=true|false (ALPHA - default=false)kube:InformerResourceVersion=true|false (ALPHA - default=false)kube:JobManagedBy=true|false (BETA - default=true)kube:JobPodReplacementPolicy=true|false (BETA - default=true)kube:KubeletCgroupDriverFromCRI=true|false (BETA - default=true)kube:KubeletCrashLoopBackOffMax=true|false (ALPHA - default=false)kube:KubeletEnsureSecretPulledImages=true|false (ALPHA - default=false)kube:KubeletFineGrainedAuthz=true|false (BETA - default=true)kube:KubeletInUserNamespace=true|false (ALPHA - default=false)kube:KubeletPSI=true|false (ALPHA - default=false)kube:KubeletPodResourcesDynamicResources=true|false (ALPHA - default=false)kube:KubeletPodResourcesGet=true|false (ALPHA - default=false)kube:KubeletSeparateDiskGC=true|false (BETA - default=true)kube:KubeletServiceAccountTokenForCredentialProviders=true|false (ALPHA - default=false)kube:KubeletTracing=true|false (BETA - default=true)kube:ListFromCacheSnapshot=true|false (ALPHA - default=false)kube:LocalStorageCapacityIsolationFSQuotaMonitoring=true|false (BETA - default=false)kube:LoggingAlphaOptions=true|false (ALPHA - default=false)kube:LoggingBetaOptions=true|false (BETA - default=true)kube:MatchLabelKeysInPodTopologySpread=true|false (BETA - default=true)kube:MaxUnavailableStatefulSet=true|false (ALPHA - default=false)kube:MemoryQoS=true|false (ALPHA - default=false)kube:MutableCSINodeAllocatableCount=true|false (ALPHA - default=false)kube:MutatingAdmissionPolicy=true|false (ALPHA - default=false)kube:NodeLogQuery=true|false (BETA - default=false)kube:NodeSwap=true|false (BETA - default=true)kube:OpenAPIEnums=true|false (BETA - default=true)kube:OrderedNamespaceDeletion=true|false (BETA - default=true)kube:PodAndContainerStatsFromCRI=true|false (ALPHA - default=false)kube:PodDeletionCost=true|false (BETA - default=true)kube:PodLevelResources=true|false (ALPHA - default=false)kube:PodLifecycleSleepAction=true|false (BETA - default=true)kube:PodLifecycleSleepActionAllowZero=true|false (BETA - default=true)kube:PodLogsQuerySplitStreams=true|false (ALPHA - default=false)kube:PodObservedGenerationTracking=true|false (ALPHA - default=false)kube:PodReadyToStartContainersCondition=true|false (BETA - default=true)kube:PodTopologyLabelsAdmission=true|false (ALPHA - default=false)kube:PortForwardWebsockets=true|false (BETA - default=true)kube:PreferSameTrafficDistribution=true|false (ALPHA - default=false)kube:ProcMountType=true|false (BETA - default=true)kube:QOSReserved=true|false (ALPHA - default=false)kube:RecoverVolumeExpansionFailure=true|false (BETA - default=true)kube:ReduceDefaultCrashLoopBackOffDecay=true|false (ALPHA - default=false)kube:RelaxedDNSSearchValidation=true|false (BETA - default=true)kube:RelaxedEnvironmentVariableValidation=true|false (BETA - default=true)kube:ReloadKubeletServerCertificateFile=true|false (BETA - default=true)kube:RemoteRequestHeaderUID=true|false (BETA - default=true)kube:ResilientWatchCacheInitialization=true|false (BETA - default=true)kube:ResourceHealthStatus=true|false (ALPHA - default=false)kube:RotateKubeletServerCertificate=true|false (BETA - default=true)kube:RuntimeClassInImageCriApi=true|false (ALPHA - default=false)kube:SELinuxChangePolicy=true|false (BETA - default=true)kube:SELinuxMount=true|false (BETA - default=false)kube:SELinuxMountReadWriteOncePod=true|false (BETA - default=true)kube:SchedulerAsyncPreemption=true|false (BETA - default=true)kube:SchedulerPopFromBackoffQ=true|false (BETA - default=true)kube:SchedulerQueueingHints=true|false (BETA - default=true)kube:SeparateTaintEvictionController=true|false (BETA - default=true)kube:ServiceAccountNodeAudienceRestriction=true|false (BETA - default=true)kube:StorageCapacityScoring=true|false (ALPHA - default=false)kube:StorageVersionAPI=true|false (ALPHA - default=false)kube:StorageVersionHash=true|false (BETA - default=true)kube:StorageVersionMigrator=true|false (ALPHA - default=false)kube:StreamingCollectionEncodingToJSON=true|false (BETA - default=true)kube:StreamingCollectionEncodingToProtobuf=true|false (BETA - default=true)kube:StrictIPCIDRValidation=true|false (ALPHA - default=false)kube:StructuredAuthenticationConfiguration=true|false (BETA - default=true)kube:SupplementalGroupsPolicy=true|false (BETA - default=true)kube:SystemdWatchdog=true|false (BETA - default=true)kube:TopologyManagerPolicyAlphaOptions=true|false (ALPHA - default=false)kube:TopologyManagerPolicyBetaOptions=true|false (BETA - default=true)kube:TranslateStreamCloseWebsocketRequests=true|false (BETA - default=true)kube:UnauthenticatedHTTP2DOSMitigation=true|false (BETA - default=true)kube:UnknownVersionInteroperabilityProxy=true|false (ALPHA - default=false)kube:UserNamespacesPodSecurityStandards=true|false (ALPHA - default=false)kube:UserNamespacesSupport=true|false (BETA - default=true)kube:VolumeAttributesClass=true|false (BETA - default=false)kube:WatchCacheInitializationPostStartHook=true|false (BETA - default=false)kube:WatchList=true|false (BETA - default=false)kube:WatchListClient=true|false (BETA - default=false)kube:WinDSR=true|false (BETA - default=true)kube:WinOverlay=true|false (BETA - default=true)kube:WindowsCPUAndMemoryAffinity=true|false (ALPHA - default=false)kube:WindowsGracefulNodeShutdown=true|false (ALPHA - default=false)--goaway-chance floatTo prevent HTTP/2 clients from getting stuck on a single apiserver, randomly close a connection (GOAWAY). The client's other in-flight requests won't be affected, and the client will reconnect, likely landing on a different apiserver after going through the load balancer again. This argument sets the fraction of requests that will be sent a GOAWAY. Clusters with single apiservers, or which don't use a load balancer, should NOT enable this. Min is 0 (off), Max is .02 (1/50 requests); .001 (1/1000) is a recommended starting point.-h, --helphelp for kube-apiserver--http2-max-streams-per-connection intThe limit that the server gives to clients for the maximum number of streams in an HTTP/2 connection. Zero means to use golang's default.--kubelet-certificate-authority stringPath to a cert file for the certificate authority.--kubelet-client-certificate stringPath to a client cert file for TLS.--kubelet-client-key stringPath to a client key file for TLS.--kubelet-preferred-address-types strings Default: "Hostname,InternalDNS,InternalIP,ExternalDNS,ExternalIP"List of the preferred NodeAddressTypes to use for kubelet connections.--kubelet-timeout duration Default: 5sTimeout for kubelet operations.--kubernetes-service-node-port intIf non-zero, the Kubernetes master service (which apiserver creates/maintains) will be of type NodePort, using this as the value of the port. If zero, the Kubernetes master service will be of type ClusterIP.--lease-reuse-duration-seconds int Default: 60The time in seconds that each lease is reused. A lower value could avoid large number of objects reusing the same lease. Notice that a too small value may cause performance problems at storage layer.--livez-grace-period durationThis option represents the maximum amount of time it should take for apiserver to complete its startup sequence and become live. From apiserver's start time to when this amount of time has elapsed, /livez will assume that unfinished post-start hooks will complete successfully and therefore return true.--log-flush-frequency duration Default: 5sMaximum number of seconds between log flushes--log-text-info-buffer-size quantity[Alpha] In text format with split output streams, the info messages can be buffered for a while to increase performance. The default value of zero bytes disables buffering. The size can be specified as number of bytes (512), multiples of 1000 (1K), multiples of 1024 (2Ki), or powers of those (3M, 4G, 5Mi, 6Gi). Enable the LoggingAlphaOptions feature gate to use this.--log-text-split-stream[Alpha] In text format, write error messages to stderr and info messages to stdout. The default is to write a single stream to stdout. Enable the LoggingAlphaOptions feature gate to use this.--logging-format string Default: "text"Sets the log format. Permitted formats: "text".--max-connection-bytes-per-sec intIf non-zero, throttle each user connection to this number of bytes/sec. Currently only applies to long-running requests.--max-mutating-requests-inflight int Default: 200This and --max-requests-inflight are summed to determine the server's total concurrency limit (which must be positive) if --enable-priority-and-fairness is true. Otherwise, this flag limits the maximum number of mutating requests in flight, or a zero value disables the limit completely.--max-requests-inflight int Default: 400This and --max-mutating-requests-inflight are summed to determine the server's total concurrency limit (which must be positive) if --enable-priority-and-fairness is true. Otherwise, this flag limits the maximum number of non-mutating requests in flight, or a zero value disables the limit completely.--min-request-timeout int Default: 1800An optional field indicating the minimum number of seconds a handler must keep a request open before timing it out. Currently only honored by the watch request handler, which picks a randomized value above this number as the connection timeout, to spread out load.--oidc-ca-file stringIf set, the OpenID server's certificate will be verified by one of the authorities in the oidc-ca-file, otherwise the host's root CA set will be used.--oidc-client-id stringThe client ID for the OpenID Connect client, must be set if oidc-issuer-url is set.--oidc-groups-claim stringIf provided, the name of a custom OpenID Connect claim for specifying user groups. The claim value is expected to be a string or array of strings. This flag is experimental, please see the authentication documentation for further details.--oidc-groups-prefix stringIf provided, all groups will be prefixed with this value to prevent conflicts with other authentication strategies.--oidc-issuer-url stringThe URL of the OpenID issuer, only HTTPS scheme will be accepted. If set, it will be used to verify the OIDC JSON Web Token (JWT).--oidc-required-claim <comma-separated 'key=value' pairs>A key=value pair that describes a required claim in the ID Token. If set, the claim is verified to be present in the ID Token with a matching value. Repeat this flag to specify multiple claims.--oidc-signing-algs strings Default: "RS256"Comma-separated list of allowed JOSE asymmetric signing algorithms. JWTs with a supported 'alg' header values are: RS256, RS384, RS512, ES256, ES384, ES512, PS256, PS384, PS512. Values are defined by RFC 7518 https://tools.ietf.org/html/rfc7518#section-3.1.--oidc-username-claim string Default: "sub"The OpenID claim to use as the user name. Note that claims other than the default ('sub') is not guaranteed to be unique and immutable. This flag is experimental, please see the authentication documentation for further details.--oidc-username-prefix stringIf provided, all usernames will be prefixed with this value. If not provided, username claims other than 'email' are prefixed by the issuer URL to avoid clashes. To skip any prefixing, provide the value '-'.--peer-advertise-ip stringIf set and the UnknownVersionInteroperabilityProxy feature gate is enabled, this IP will be used by peer kube-apiservers to proxy requests to this kube-apiserver when the request cannot be handled by the peer due to version skew between the kube-apiservers. This flag is only used in clusters configured with multiple kube-apiservers for high availability.--peer-advertise-port stringIf set and the UnknownVersionInteroperabilityProxy feature gate is enabled, this port will be used by peer kube-apiservers to proxy requests to this kube-apiserver when the request cannot be handled by the peer due to version skew between the kube-apiservers. This flag is only used in clusters configured with multiple kube-apiservers for high availability.--peer-ca-file stringIf set and the UnknownVersionInteroperabilityProxy feature gate is enabled, this file will be used to verify serving certificates of peer kube-apiservers. This flag is only used in clusters configured with multiple kube-apiservers for high availability.--permit-address-sharingIf true, SO_REUSEADDR will be used when binding the port. This allows binding to wildcard IPs like 0.0.0.0 and specific IPs in parallel, and it avoids waiting for the kernel to release sockets in TIME_WAIT state. [default=false]--permit-port-sharingIf true, SO_REUSEPORT will be used when binding the port, which allows more than one instance to bind on the same address and port. [default=false]--profiling Default: trueEnable profiling via web interface host:port/debug/pprof/--proxy-client-cert-file stringClient certificate used to prove the identity of the aggregator or kube-apiserver when it must call out during a request. This includes proxying requests to a user api-server and calling out to webhook admission plugins. It is expected that this cert includes a signature from the CA in the --requestheader-client-ca-file flag. That CA is published in the 'extension-apiserver-authentication' configmap in the kube-system namespace. Components receiving calls from kube-aggregator should use that CA to perform their half of the mutual TLS verification.--proxy-client-key-file stringPrivate key for the client certificate used to prove the identity of the aggregator or kube-apiserver when it must call out during a request. This includes proxying requests to a user api-server and calling out to webhook admission plugins.--request-timeout duration Default: 1m0sAn optional field indicating the duration a handler must keep a request open before timing it out. This is the default request timeout for requests but may be overridden by flags such as --min-request-timeout for specific types of requests.--requestheader-allowed-names stringsList of client certificate common names to allow to provide usernames in headers specified by --requestheader-username-headers. If empty, any client certificate validated by the authorities in --requestheader-client-ca-file is allowed.--requestheader-client-ca-file stringRoot certificate bundle to use to verify client certificates on incoming requests before trusting usernames in headers specified by --requestheader-username-headers. WARNING: generally do not depend on authorization being already done for incoming requests.--requestheader-extra-headers-prefix stringsList of request header prefixes to inspect. X-Remote-Extra- is suggested.--requestheader-group-headers stringsList of request headers to inspect for groups. X-Remote-Group is suggested.--requestheader-uid-headers stringsList of request headers to inspect for UIDs. X-Remote-Uid is suggested. Requires the RemoteRequestHeaderUID feature to be enabled.--requestheader-username-headers stringsList of request headers to inspect for usernames. X-Remote-User is common.--runtime-config <comma-separated 'key=value' pairs>A set of key=value pairs that enable or disable built-in APIs. Supported options are:v1=true|false for the core API group<group>/<version>=true|false for a specific API group and version (e.g. apps/v1=true)api/all=true|false controls all API versionsapi/ga=true|false controls all API versions of the form v[0-9]+api/beta=true|false controls all API versions of the form v[0-9]+beta[0-9]+api/alpha=true|false controls all API versions of the form v[0-9]+alpha[0-9]+api/legacy is deprecated, and will be removed in a future version--runtime-config-emulation-forward-compatibleIf true, APIs identified by group/version that are enabled in the --runtime-config flag will be installed even if it is introduced after the emulation version. If false, server would fail to start if any APIs identified by group/version that are enabled in the --runtime-config flag are introduced after the emulation version. Can only be set to true if the emulation version is lower than the binary version.--secure-port int Default: 6443The port on which to serve HTTPS with authentication and authorization. It cannot be switched off with 0.--service-account-extend-token-expiration Default: trueTurns on projected service account expiration extension during token generation, which helps safe transition from legacy token to bound service account token feature. If this flag is enabled, admission injected tokens would be extended up to 1 year to prevent unexpected failure during transition, ignoring value of service-account-max-token-expiration.--service-account-issuer stringsIdentifier of the service account token issuer. The issuer will assert this identifier in "iss" claim of issued tokens. This value is a string or URI. If this option is not a valid URI per the OpenID Discovery 1.0 spec, the ServiceAccountIssuerDiscovery feature will remain disabled, even if the feature gate is set to true. It is highly recommended that this value comply with the OpenID spec: https://openid.net/specs/openid-connect-discovery-1_0.html. In practice, this means that service-account-issuer must be an https URL. It is also highly recommended that this URL be capable of serving OpenID discovery documents at {service-account-issuer}/.well-known/openid-configuration. When this flag is specified multiple times, the first is used to generate tokens and all are used to determine which issuers are accepted.--service-account-jwks-uri stringOverrides the URI for the JSON Web Key Set in the discovery doc served at /.well-known/openid-configuration. This flag is useful if the discovery docand key set are served to relying parties from a URL other than the API server's external (as auto-detected or overridden with external-hostname).--service-account-key-file stringsFile containing PEM-encoded x509 RSA or ECDSA private or public keys, used to verify ServiceAccount tokens. The specified file can contain multiple keys, and the flag can be specified multiple times with different files. If unspecified, --tls-private-key-file is used. Must be specified when --service-account-signing-key-file is provided--service-account-lookup Default: trueIf true, validate ServiceAccount tokens exist in etcd as part of authentication.--service-account-max-token-expiration durationThe maximum validity duration of a token created by the service account token issuer. If an otherwise valid TokenRequest with a validity duration larger than this value is requested, a token will be issued with a validity duration of this value.--service-account-signing-endpoint stringPath to socket where a external JWT signer is listening. This flag is mutually exclusive with --service-account-signing-key-file and --service-account-key-file. Requires enabling feature gate (ExternalServiceAccountTokenSigner)--service-account-signing-key-file stringPath to the file that contains the current private key of the service account token issuer. The issuer will sign issued ID tokens with this private key.--service-cluster-ip-range stringA CIDR notation IP range from which to assign service cluster IPs. This must not overlap with any IP ranges assigned to nodes or pods. Max of two dual-stack CIDRs is allowed.--service-node-port-range <a string in the form 'N1-N2'> Default: 30000-32767A port range to reserve for services with NodePort visibility. This must not overlap with the ephemeral port range on nodes. Example: '30000-32767'. Inclusive at both ends of the range.--show-hidden-metrics-for-version stringThe previous version for which you want to show hidden metrics. Only the previous minor version is meaningful, other values will not be allowed. The format is <major>.<minor>, e.g.: '1.16'. The purpose of this format is make sure you have the opportunity to notice if the next release hides additional metrics, rather than being surprised when they are permanently removed in the release after that.--shutdown-delay-duration durationTime to delay the termination. During that time the server keeps serving requests normally. The endpoints /healthz and /livez will return success, but /readyz immediately returns failure. Graceful termination starts after this delay has elapsed. This can be used to allow load balancer to stop sending traffic to this server.--shutdown-send-retry-afterIf true the HTTP Server will continue listening until all non long running request(s) in flight have been drained, during this window all incoming requests will be rejected with a status code 429 and a 'Retry-After' response header, in addition 'Connection: close' response header is set in order to tear down the TCP connection when idle.--shutdown-watch-termination-grace-period durationThis option, if set, represents the maximum amount of grace period the apiserver will wait for active watch request(s) to drain during the graceful server shutdown window.--storage-backend stringThe storage backend for persistence. Options: 'etcd3' (default).--storage-initialization-timeout duration Default: 1m0sMaximum amount of time to wait for storage initialization before declaring apiserver ready. Defaults to 1m.--storage-media-type string Default: "application/vnd.kubernetes.protobuf"The media type to use to store objects in storage. Some resources or storage backends may only support a specific media type and will ignore this setting. Supported media types: [application/json, application/yaml, application/vnd.kubernetes.protobuf]--strict-transport-security-directives stringsList of directives for HSTS, comma separated. If this list is empty, then HSTS directives will not be added. Example: 'max-age=31536000,includeSubDomains,preload'--tls-cert-file stringFile containing the default x509 Certificate for HTTPS. (CA cert, if any, concatenated after server cert). If HTTPS serving is enabled, and --tls-cert-file and --tls-private-key-file are not provided, a self-signed certificate and key are generated for the public address and saved to the directory specified by --cert-dir.--tls-cipher-suites stringsComma-separated list of cipher suites for the server. If omitted, the default Go cipher suites will be used.Preferred values: TLS_AES_128_GCM_SHA256, TLS_AES_256_GCM_SHA384, TLS_CHACHA20_POLY1305_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305, TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305, TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256.Insecure values: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_RC4_128_SHA, TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_RC4_128_SHA, TLS_RSA_WITH_3DES_EDE_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_256_GCM_SHA384, TLS_RSA_WITH_RC4_128_SHA.--tls-min-version stringMinimum TLS version supported. Possible values: VersionTLS10, VersionTLS11, VersionTLS12, VersionTLS13--tls-private-key-file stringFile containing the default x509 private key matching --tls-cert-file.--tls-sni-cert-key stringA pair of x509 certificate and private key file paths, optionally suffixed with a list of domain patterns which are fully qualified domain names, possibly with prefixed wildcard segments. The domain patterns also allow IP addresses, but IPs should only be used if the apiserver has visibility to the IP address requested by a client. If no domain patterns are provided, the names of the certificate are extracted. Non-wildcard matches trump over wildcard matches, explicit domain patterns trump over extracted names. For multiple key/certificate pairs, use the --tls-sni-cert-key multiple times. Examples: "example.crt,example.key" or "foo.crt,foo.key:*.foo.com,foo.com".--token-auth-file stringIf set, the file that will be used to secure the secure port of the API server via token authentication.--tracing-config-file stringFile with apiserver tracing configuration.-v, --v intnumber for the log level verbosity--version version[=true]--version, --version=raw prints version information and quits; --version=vX.Y.Z... sets the reported version--vmodule pattern=N,...comma-separated list of pattern=N settings for file-filtered logging (only works for text log format)--watch-cache Default: trueEnable watch caching in the apiserver--watch-cache-sizes stringsWatch cache size settings for some resources (pods, nodes, etc.), comma separated. The individual setting format: resource[.group]#size, where resource is lowercase plural (no version), group is omitted for resources of apiVersion v1 (the legacy core API) and included for others, and size is a number. This option is only meaningful for resources built into the apiserver, not ones defined by CRDs or aggregated from external servers, and is only consulted if the watch-cache is enabled. The only meaningful size setting to supply here is zero, which means to disable watch caching for the associated resource; all non-zero values are equivalent and mean to not disable watch caching for that resourceThis page is automatically generated.If you plan to report an issue with this page, mention that the page is auto-generated in your issue description. The fix may need to happen elsewhere in the Kubernetes project.


# Automatic Cleanup for Finished Jobs

Automatic Cleanup for Finished JobsA time-to-live mechanism to clean up old Jobs that have finished execution.FEATURE STATE: Kubernetes v1.23 [stable]When your Job has finished, it's useful to keep that Job in the API (and not immediately delete the Job) so that you can tell whether the Job succeeded or failed.Kubernetes' TTL-after-finished controller provides a TTL (time to live) mechanism to limit the lifetime of Job objects that have finished execution.Cleanup for finished JobsThe TTL-after-finished controller is only supported for Jobs. You can use this mechanism to clean up finished Jobs (either Complete or Failed) automatically by specifying the .spec.ttlSecondsAfterFinished field of a Job, as in this example.The TTL-after-finished controller assumes that a Job is eligible to be cleaned up TTL seconds after the Job has finished. The timer starts once the status condition of the Job changes to show that the Job is either Complete or Failed; once the TTL has expired, that Job becomes eligible for cascading removal. When the TTL-after-finished controller cleans up a job, it will delete it cascadingly, that is to say it will delete its dependent objects together with it.Kubernetes honors object lifecycle guarantees on the Job, such as waiting for finalizers.You can set the TTL seconds at any time. Here are some examples for setting the .spec.ttlSecondsAfterFinished field of a Job:Specify this field in the Job manifest, so that a Job can be cleaned up automatically some time after it finishes.Manually set this field of existing, already finished Jobs, so that they become eligible for cleanup.Use a mutating admission webhook to set this field dynamically at Job creation time. Cluster administrators can use this to enforce a TTL policy for finished jobs.Use a mutating admission webhook to set this field dynamically after the Job has finished, and choose different TTL values based on job status, labels. For this case, the webhook needs to detect changes to the .status of the Job and only set a TTL when the Job is being marked as completed.Write your own controller to manage the cleanup TTL for Jobs that match a particular selector.CaveatsUpdating TTL for finished JobsYou can modify the TTL period, e.g. .spec.ttlSecondsAfterFinished field of Jobs, after the job is created or has finished. If you extend the TTL period after the existing ttlSecondsAfterFinished period has expired, Kubernetes doesn't guarantee to retain that Job, even if an update to extend the TTL returns a successful API response.Time skewBecause the TTL-after-finished controller uses timestamps stored in the Kubernetes jobs to determine whether the TTL has expired or not, this feature is sensitive to time skew in your cluster, which may cause the control plane to clean up Job objects at the wrong time.Clocks aren't always correct, but the difference should be very small. Please be aware of this risk when setting a non-zero TTL.What's nextRead Clean up Jobs automaticallyRefer to the Kubernetes Enhancement Proposal (KEP) for adding this mechanism.


# kubeadm config

kubeadm configDuring kubeadm init, kubeadm uploads the ClusterConfiguration object to your cluster in a ConfigMap called kubeadm-config in the kube-system namespace. This configuration is then read during kubeadm join, kubeadm reset and kubeadm upgrade.You can use kubeadm config print to print the default static configuration that kubeadm uses for kubeadm init and kubeadm join.Note:The output of the command is meant to serve as an example. You must manually edit the output of this command to adapt to your setup. Remove the fields that you are not certain about and kubeadm will try to default them on runtime by examining the host.For more information on init and join navigate to Using kubeadm init with a configuration file or Using kubeadm join with a configuration file.For more information on using the kubeadm configuration API navigate to Customizing components with the kubeadm API.You can use kubeadm config migrate to convert your old configuration files that contain a deprecated API version to a newer, supported API version.kubeadm config validate can be used for validating a configuration file.kubeadm config images list and kubeadm config images pull can be used to list and pull the images that kubeadm requires.kubeadm config printPrint configurationSynopsisThis command prints configurations for subcommands provided. For details, see: https://pkg.go.dev/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm#section-directorieskubeadm config print [flags] Options-h, --helphelp for printOptions inherited from parent commands--kubeconfig string Default: "/etc/kubernetes/admin.conf"The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations can be searched for an existing kubeconfig file.--rootfs stringThe path to the 'real' host root filesystem. This will cause kubeadm to chroot into the provided path.kubeadm config print init-defaultsPrint default init configuration, that can be used for 'kubeadm init'SynopsisThis command prints objects such as the default init configuration that is used for 'kubeadm init'.Note that sensitive values like the Bootstrap Token fields are replaced with placeholder values like "abcdef.0123456789abcdef" in order to pass validation but not perform the real computation for creating a token.kubeadm config print init-defaults [flags] Options--component-configs stringsA comma-separated list for component config API objects to print the default values for. Available values: [KubeProxyConfiguration KubeletConfiguration]. If this flag is not set, no component configs will be printed.-h, --helphelp for init-defaultsOptions inherited from parent commands--kubeconfig string Default: "/etc/kubernetes/admin.conf"The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations can be searched for an existing kubeconfig file.--rootfs stringThe path to the 'real' host root filesystem. This will cause kubeadm to chroot into the provided path.kubeadm config print join-defaultsPrint default join configuration, that can be used for 'kubeadm join'SynopsisThis command prints objects such as the default join configuration that is used for 'kubeadm join'.Note that sensitive values like the Bootstrap Token fields are replaced with placeholder values like "abcdef.0123456789abcdef" in order to pass validation but not perform the real computation for creating a token.kubeadm config print join-defaults [flags] Options-h, --helphelp for join-defaultsOptions inherited from parent commands--kubeconfig string Default: "/etc/kubernetes/admin.conf"The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations can be searched for an existing kubeconfig file.--rootfs stringThe path to the 'real' host root filesystem. This will cause kubeadm to chroot into the provided path.kubeadm config migrateRead an older version of the kubeadm configuration API types from a file, and output the similar config object for the newer versionSynopsisThis command lets you convert configuration objects of older versions to the latest supported version, locally in the CLI tool without ever touching anything in the cluster. In this version of kubeadm, the following API versions are supported:kubeadm.k8s.io/v1beta4Further, kubeadm can only write out config of version "kubeadm.k8s.io/v1beta4", but read both types. So regardless of what version you pass to the --old-config parameter here, the API object will be read, deserialized, defaulted, converted, validated, and re-serialized when written to stdout or --new-config if specified.In other words, the output of this command is what kubeadm actually would read internally if you submitted this file to "kubeadm init"kubeadm config migrate [flags] Options--allow-experimental-apiAllow migration to experimental, unreleased APIs.-h, --helphelp for migrate--new-config stringPath to the resulting equivalent kubeadm config file using the new API version. Optional, if not specified output will be sent to STDOUT.--old-config stringPath to the kubeadm config file that is using an old API version and should be converted. This flag is mandatory.Options inherited from parent commands--kubeconfig string Default: "/etc/kubernetes/admin.conf"The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations can be searched for an existing kubeconfig file.--rootfs stringThe path to the 'real' host root filesystem. This will cause kubeadm to chroot into the provided path.kubeadm config validateRead a file containing the kubeadm configuration API and report any validation problemsSynopsisThis command lets you validate a kubeadm configuration API file and report any warnings and errors. If there are no errors the exit status will be zero, otherwise it will be non-zero. Any unmarshaling problems such as unknown API fields will trigger errors. Unknown API versions and fields with invalid values will also trigger errors. Any other errors or warnings may be reported depending on contents of the input file.In this version of kubeadm, the following API versions are supported:kubeadm.k8s.io/v1beta4kubeadm config validate [flags] Options--allow-experimental-apiAllow validation of experimental, unreleased APIs.--config stringPath to a kubeadm configuration file.-h, --helphelp for validateOptions inherited from parent commands--kubeconfig string Default: "/etc/kubernetes/admin.conf"The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations can be searched for an existing kubeconfig file.--rootfs stringThe path to the 'real' host root filesystem. This will cause kubeadm to chroot into the provided path.kubeadm config images listSynopsisPrint a list of images kubeadm will use. The configuration file is used in case any images or image repositories are customizedkubeadm config images list [flags] Options--allow-missing-template-keys Default: trueIf true, ignore any errors in templates when a field or map key is missing in the template. Only applies to golang and jsonpath output formats.--config stringPath to a kubeadm configuration file.--feature-gates stringA set of key=value pairs that describe feature gates for various features. Options are:ControlPlaneKubeletLocalMode=true|false (BETA - default=true)NodeLocalCRISocket=true|false (BETA - default=true)PublicKeysECDSA=true|false (DEPRECATED - default=false)RootlessControlPlane=true|false (ALPHA - default=false)WaitForAllControlPlaneComponents=true|false (default=true)-h, --helphelp for list--image-repository string Default: "registry.k8s.io"Choose a container registry to pull control plane images from--kubernetes-version string Default: "stable-1"Choose a specific Kubernetes version for the control plane.-o, --output string Default: "text"Output format. One of: text|json|yaml|go-template|go-template-file|template|templatefile|jsonpath|jsonpath-as-json|jsonpath-file.--show-managed-fieldsIf true, keep the managedFields when printing objects in JSON or YAML format.Options inherited from parent commands--kubeconfig string Default: "/etc/kubernetes/admin.conf"The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations can be searched for an existing kubeconfig file.--rootfs stringThe path to the 'real' host root filesystem. This will cause kubeadm to chroot into the provided path.kubeadm config images pullSynopsisPull images used by kubeadmkubeadm config images pull [flags] Options--config stringPath to a kubeadm configuration file.--cri-socket stringPath to the CRI socket to connect. If empty kubeadm will try to auto-detect this value; use this option only if you have more than one CRI installed or if you have non-standard CRI socket.--feature-gates stringA set of key=value pairs that describe feature gates for various features. Options are:ControlPlaneKubeletLocalMode=true|false (BETA - default=true)NodeLocalCRISocket=true|false (BETA - default=true)PublicKeysECDSA=true|false (DEPRECATED - default=false)RootlessControlPlane=true|false (ALPHA - default=false)WaitForAllControlPlaneComponents=true|false (default=true)-h, --helphelp for pull--image-repository string Default: "registry.k8s.io"Choose a container registry to pull control plane images from--kubernetes-version string Default: "stable-1"Choose a specific Kubernetes version for the control plane.Options inherited from parent commands--kubeconfig string Default: "/etc/kubernetes/admin.conf"The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations can be searched for an existing kubeconfig file.--rootfs stringThe path to the 'real' host root filesystem. This will cause kubeadm to chroot into the provided path.What's nextkubeadm upgrade to upgrade a Kubernetes cluster to a newer version


# Kubeadm

KubeadmKubeadm is a tool built to provide kubeadm init and kubeadm join as best-practice "fast paths" for creating Kubernetes clusters.kubeadm performs the actions necessary to get a minimum viable cluster up and running. By design, it cares only about bootstrapping, not about provisioning machines. Likewise, installing various nice-to-have addons, like the Kubernetes Dashboard, monitoring solutions, and cloud-specific addons, is not in scope.Instead, we expect higher-level and more tailored tooling to be built on top of kubeadm, and ideally, using kubeadm as the basis of all deployments will make it easier to create conformant clusters.How to installTo install kubeadm, see the installation guide.What's nextkubeadm init to bootstrap a Kubernetes control-plane nodekubeadm join to bootstrap a Kubernetes worker node and join it to the clusterkubeadm upgrade to upgrade a Kubernetes cluster to a newer versionkubeadm config if you initialized your cluster using kubeadm v1.7.x or lower, to configure your cluster for kubeadm upgradekubeadm token to manage tokens for kubeadm joinkubeadm reset to revert any changes made to this host by kubeadm init or kubeadm joinkubeadm certs to manage Kubernetes certificateskubeadm kubeconfig to manage kubeconfig fileskubeadm version to print the kubeadm versionkubeadm alpha to preview a set of features made available for gathering


# Implementation details

Implementation detailsFEATURE STATE: Kubernetes v1.10 [stable]kubeadm init and kubeadm join together provide a nice user experience for creating a bare Kubernetes cluster from scratch, that aligns with the best-practices. However, it might not be obvious how kubeadm does that.This document provides additional details on what happens under the hood, with the aim of sharing knowledge on the best practices for a Kubernetes cluster.Core design principlesThe cluster that kubeadm init and kubeadm join set up should be:Secure: It should adopt latest best-practices like:enforcing RBACusing the Node Authorizerusing secure communication between the control plane componentsusing secure communication between the API server and the kubeletslock-down the kubelet APIlocking down access to the API for system components like the kube-proxy and CoreDNSlocking down what a Bootstrap Token can accessUser-friendly: The user should not have to run anything more than a couple of commands:kubeadm initexport KUBECONFIG=/etc/kubernetes/admin.confkubectl apply -f <network-plugin-of-choice.yaml>kubeadm join --token <token> <endpoint>:<port>Extendable:It should not favor any particular network provider. Configuring the cluster network is out-of-scopeIt should provide the possibility to use a config file for customizing various parametersConstants and well-known values and pathsIn order to reduce complexity and to simplify development of higher level tools that build on top of kubeadm, it uses a limited set of constant values for well-known paths and file names.The Kubernetes directory /etc/kubernetes is a constant in the application, since it is clearly the given path in a majority of cases, and the most intuitive location; other constant paths and file names are:/etc/kubernetes/manifests as the path where the kubelet should look for static Pod manifests. Names of static Pod manifests are:etcd.yamlkube-apiserver.yamlkube-controller-manager.yamlkube-scheduler.yaml/etc/kubernetes/ as the path where kubeconfig files with identities for control plane components are stored. Names of kubeconfig files are:kubelet.conf (bootstrap-kubelet.conf during TLS bootstrap)controller-manager.confscheduler.confadmin.conf for the cluster admin and kubeadm itselfsuper-admin.conf for the cluster super-admin that can bypass RBACNames of certificates and key files:ca.crt, ca.key for the Kubernetes certificate authorityapiserver.crt, apiserver.key for the API server certificateapiserver-kubelet-client.crt, apiserver-kubelet-client.key for the client certificate used by the API server to connect to the kubelets securelysa.pub, sa.key for the key used by the controller manager when signing ServiceAccountfront-proxy-ca.crt, front-proxy-ca.key for the front proxy certificate authorityfront-proxy-client.crt, front-proxy-client.key for the front proxy clientThe kubeadm configuration file formatMost kubeadm commands support a --config flag which allows passing a configuration file from disk. The configuration file format follows the common Kubernetes API apiVersion / kind scheme, but is considered a component configuration format. Several Kubernetes components, such as the kubelet, also support file-based configuration.Different kubeadm subcommands require a different kind of configuration file. For example, InitConfiguration for kubeadm init, JoinConfiguration for kubeadm join, UpgradeConfiguration for kubeadm upgrade and ResetConfiguration for kubeadm reset.The command kubeadm config migrate can be used to migrate an older format configuration file to a newer (current) configuration format. The kubeadm tool only supports migrating from deprecated configuration formats to the current format.See the kubeadm configuration reference page for more details.kubeadm init workflow internal designThe kubeadm init consists of a sequence of atomic work tasks to perform, as described in the kubeadm init internal workflow.The kubeadm init phase command allows users to invoke each task individually, and ultimately offers a reusable and composable API/toolbox that can be used by other Kubernetes bootstrap tools, by any IT automation tool or by an advanced user for creating custom clusters.Preflight checksKubeadm executes a set of preflight checks before starting the init, with the aim to verify preconditions and avoid common cluster startup problems. The user can skip specific preflight checks or all of them with the --ignore-preflight-errors option.[Warning] if the Kubernetes version to use (specified with the --kubernetes-version flag) is at least one minor version higher than the kubeadm CLI version.Kubernetes system requirements:if running on linux:[Error] if Kernel is older than the minimum required version[Error] if required cgroups subsystem aren't set up[Error] if the CRI endpoint does not answer[Error] if user is not root[Error] if the machine hostname is not a valid DNS subdomain[Warning] if the host name cannot be reached via network lookup[Error] if kubelet version is lower that the minimum kubelet version supported by kubeadm (current minor -1)[Error] if kubelet version is at least one minor higher than the required controlplane version (unsupported version skew)[Warning] if kubelet service does not exist or if it is disabled[Warning] if firewalld is active[Error] if API server bindPort or ports 10250/10251/10252 are used[Error] if /etc/kubernetes/manifest folder already exists and it is not empty[Error] if swap is on[Error] if ip, iptables, mount, nsenter commands are not present in the command path[Warning] if ethtool, tc, touch commands are not present in the command path[Warning] if extra arg flags for API server, controller manager, scheduler contains some invalid options[Warning] if connection to https://API.AdvertiseAddress:API.BindPort goes through proxy[Warning] if connection to services subnet goes through proxy (only first address checked)[Warning] if connection to Pods subnet goes through proxy (only first address checked)If external etcd is provided:[Error] if etcd version is older than the minimum required version[Error] if etcd certificates or keys are specified, but not providedIf external etcd is NOT provided (and thus local etcd will be installed):[Error] if ports 2379 is used[Error] if Etcd.DataDir folder already exists and it is not emptyIf authorization mode is ABAC:[Error] if abac_policy.json does not existIf authorization mode is WebHook[Error] if webhook_authz.conf does not existNote:Preflight checks can be invoked individually with the kubeadm init phase preflight command.Generate the necessary certificatesKubeadm generates certificate and private key pairs for different purposes:A self signed certificate authority for the Kubernetes cluster saved into ca.crt file and ca.key private key fileA serving certificate for the API server, generated using ca.crt as the CA, and saved into apiserver.crt file with its private key apiserver.key. This certificate should contain the following alternative names:The Kubernetes service's internal clusterIP (the first address in the services CIDR, e.g. 10.96.0.1 if service subnet is 10.96.0.0/12)Kubernetes DNS names, e.g. kubernetes.default.svc.cluster.local if --service-dns-domain flag value is cluster.local, plus default DNS names kubernetes.default.svc, kubernetes.default, kubernetesThe node-nameThe --apiserver-advertise-addressAdditional alternative names specified by the userA client certificate for the API server to connect to the kubelets securely, generated using ca.crt as the CA and saved into apiserver-kubelet-client.crt file with its private key apiserver-kubelet-client.key. This certificate should be in the system:masters organizationA private key for signing ServiceAccount Tokens saved into sa.key file along with its public key sa.pubA certificate authority for the front proxy saved into front-proxy-ca.crt file with its key front-proxy-ca.keyA client certificate for the front proxy client, generated using front-proxy-ca.crt as the CA and saved into front-proxy-client.crt file with its private keyfront-proxy-client.keyCertificates are stored by default in /etc/kubernetes/pki, but this directory is configurable using the --cert-dir flag.Please note that:If a given certificate and private key pair both exist, and their content is evaluated to be compliant with the above specs, the existing files will be used and the generation phase for the given certificate will be skipped. This means the user can, for example, copy an existing CA to /etc/kubernetes/pki/ca.{crt,key}, and then kubeadm will use those files for signing the rest of the certs. See also using custom certificatesFor the CA, it is possible to provide the ca.crt file but not the ca.key file. If all other certificates and kubeconfig files are already in place, kubeadm recognizes this condition and activates the ExternalCA, which also implies the csrsigner controller in controller-manager won't be startedIf kubeadm is running in external CA mode; all the certificates must be provided by the user, because kubeadm cannot generate them by itselfIn case kubeadm is executed in the --dry-run mode, certificate files are written in a temporary folderCertificate generation can be invoked individually with the kubeadm init phase certs all commandGenerate kubeconfig files for control plane componentsKubeadm generates kubeconfig files with identities for control plane components:A kubeconfig file for the kubelet to use during TLS bootstrap - /etc/kubernetes/bootstrap-kubelet.conf. Inside this file, there is a bootstrap-token or embedded client certificates for authenticating this node with the cluster.This client certificate should:Be in the system:nodes organization, as required by the Node Authorization moduleHave the Common Name (CN) system:node:<hostname-lowercased>A kubeconfig file for controller-manager, /etc/kubernetes/controller-manager.conf; inside this file is embedded a client certificate with controller-manager identity. This client certificate should have the CN system:kube-controller-manager, as defined by default RBAC core components rolesA kubeconfig file for scheduler, /etc/kubernetes/scheduler.conf; inside this file is embedded a client certificate with scheduler identity. This client certificate should have the CN system:kube-scheduler, as defined by default RBAC core components rolesAdditionally, a kubeconfig file for kubeadm as an administrative entity is generated and stored in /etc/kubernetes/admin.conf. This file includes a certificate with Subject: O = kubeadm:cluster-admins, CN = kubernetes-admin. kubeadm:cluster-admins is a group managed by kubeadm. It is bound to the cluster-admin ClusterRole during kubeadm init, by using the super-admin.conf file, which does not require RBAC. This admin.conf file must remain on control plane nodes and should not be shared with additional users.During kubeadm init another kubeconfig file is generated and stored in /etc/kubernetes/super-admin.conf. This file includes a certificate with Subject: O = system:masters, CN = kubernetes-super-admin. system:masters is a superuser group that bypasses RBAC and makes super-admin.conf useful in case of an emergency where a cluster is locked due to RBAC misconfiguration. The super-admin.conf file must be stored in a safe location and should not be shared with additional users.See RBAC user facing role bindings for additional information on RBAC and built-in ClusterRoles and groups.You can run kubeadm kubeconfig user to generate kubeconfig files for additional users.Caution:The generated configuration files include an embedded authentication key, and you should treat them as confidential.Also note that:ca.crt certificate is embedded in all the kubeconfig files.If a given kubeconfig file exists, and its content is evaluated as compliant with the above specs, the existing file will be used and the generation phase for the given kubeconfig will be skippedIf kubeadm is running in ExternalCA mode, all the required kubeconfig must be provided by the user as well, because kubeadm cannot generate any of them by itselfIn case kubeadm is executed in the --dry-run mode, kubeconfig files are written in a temporary folderGeneration of kubeconfig files can be invoked individually with the kubeadm init phase kubeconfig all commandGenerate static Pod manifests for control plane componentsKubeadm writes static Pod manifest files for control plane components to /etc/kubernetes/manifests. The kubelet watches this directory for Pods to be created on startup.Static Pod manifests share a set of common properties:All static Pods are deployed on kube-system namespaceAll static Pods get tier:control-plane and component:{component-name} labelsAll static Pods use the system-node-critical priority classhostNetwork: true is set on all static Pods to allow control plane startup before a network is configured; as a consequence:The address that the controller-manager and the scheduler use to refer to the API server is 127.0.0.1If the etcd server is set up locally, the etcd-server address will be set to 127.0.0.1:2379Leader election is enabled for both the controller-manager and the schedulerController-manager and the scheduler will reference kubeconfig files with their respective, unique identitiesAll static Pods get any extra flags or patches that you specify, as described in passing custom arguments to control plane componentsAll static Pods get any extra Volumes specified by the user (Host path)Please note that:All images will be pulled from registry.k8s.io by default. See using custom images for customizing the image repositoryIn case kubeadm is executed in the --dry-run mode, static Pod files are written in a temporary folderStatic Pod manifest generation for control plane components can be invoked individually with the kubeadm init phase control-plane all commandAPI serverThe static Pod manifest for the API server is affected by the following parameters provided by the users:The apiserver-advertise-address and apiserver-bind-port to bind to; if not provided, those values default to the IP address of the default network interface on the machine and port 6443The service-cluster-ip-range to use for servicesIf an external etcd server is specified, the etcd-servers address and related TLS settings (etcd-cafile, etcd-certfile, etcd-keyfile); if an external etcd server is not provided, a local etcd will be used (via host network)If a cloud provider is specified, the corresponding --cloud-provider parameter is configured together with the --cloud-config path if such file exists (this is experimental, alpha and will be removed in a future version)Other API server flags that are set unconditionally are:--insecure-port=0 to avoid insecure connections to the api server--enable-bootstrap-token-auth=true to enable the BootstrapTokenAuthenticator authentication module. See TLS Bootstrapping for more details--allow-privileged to true (required e.g. by kube proxy)--requestheader-client-ca-file to front-proxy-ca.crt--enable-admission-plugins to:NamespaceLifecycle e.g. to avoid deletion of system reserved namespacesLimitRanger and ResourceQuota to enforce limits on namespacesServiceAccount to enforce service account automationPersistentVolumeLabel attaches region or zone labels to PersistentVolumes as defined by the cloud provider (This admission controller is deprecated and will be removed in a future version. It is not deployed by kubeadm by default with v1.9 onwards when not explicitly opting into using gce or aws as cloud providers)DefaultStorageClass to enforce default storage class on PersistentVolumeClaim objectsDefaultTolerationSecondsNodeRestriction to limit what a kubelet can modify (e.g. only pods on this node)--kubelet-preferred-address-types to InternalIP,ExternalIP,Hostname; this makes kubectl logs and other API server-kubelet communication work in environments where the hostnames of the nodes aren't resolvableFlags for using certificates generated in previous steps:--client-ca-file to ca.crt--tls-cert-file to apiserver.crt--tls-private-key-file to apiserver.key--kubelet-client-certificate to apiserver-kubelet-client.crt--kubelet-client-key to apiserver-kubelet-client.key--service-account-key-file to sa.pub--requestheader-client-ca-file to front-proxy-ca.crt--proxy-client-cert-file to front-proxy-client.crt--proxy-client-key-file to front-proxy-client.keyOther flags for securing the front proxy (API Aggregation) communications:--requestheader-username-headers=X-Remote-User--requestheader-group-headers=X-Remote-Group--requestheader-extra-headers-prefix=X-Remote-Extra---requestheader-allowed-names=front-proxy-clientController managerThe static Pod manifest for the controller manager is affected by following parameters provided by the users:If kubeadm is invoked specifying a --pod-network-cidr, the subnet manager feature required for some CNI network plugins is enabled by setting:--allocate-node-cidrs=true--cluster-cidr and --node-cidr-mask-size flags according to the given CIDROther flags that are set unconditionally are:--controllers enabling all the default controllers plus BootstrapSigner and TokenCleaner controllers for TLS bootstrap. See TLS Bootstrapping for more details.--use-service-account-credentials to trueFlags for using certificates generated in previous steps:--root-ca-file to ca.crt--cluster-signing-cert-file to ca.crt, if External CA mode is disabled, otherwise to ""--cluster-signing-key-file to ca.key, if External CA mode is disabled, otherwise to ""--service-account-private-key-file to sa.keySchedulerThe static Pod manifest for the scheduler is not affected by parameters provided by the user.Generate static Pod manifest for local etcdIf you specified an external etcd, this step will be skipped, otherwise kubeadm generates a static Pod manifest file for creating a local etcd instance running in a Pod with following attributes:listen on localhost:2379 and use HostNetwork=truemake a hostPath mount out from the dataDir to the host's filesystemAny extra flags specified by the userPlease note that:The etcd container image will be pulled from registry.gcr.io by default. See using custom images for customizing the image repository.If you run kubeadm in --dry-run mode, the etcd static Pod manifest is written into a temporary folder.You can directly invoke static Pod manifest generation for local etcd, using the kubeadm init phase etcd local command.Wait for the control plane to come upOn control plane nodes, kubeadm waits up to 4 minutes for the control plane components and the kubelet to be available. It does that by performing a health check on the respective component /healthz or /livez endpoints.After the control plane is up, kubeadm completes the tasks described in following paragraphs.Save the kubeadm ClusterConfiguration in a ConfigMap for later referencekubeadm saves the configuration passed to kubeadm init in a ConfigMap named kubeadm-config under kube-system namespace.This will ensure that kubeadm actions executed in future (e.g kubeadm upgrade) will be able to determine the actual/current cluster state and make new decisions based on that data.Please note that:Before saving the ClusterConfiguration, sensitive information like the token is stripped from the configurationUpload of control plane node configuration can be invoked individually with the command kubeadm init phase upload-config.Mark the node as control-planeAs soon as the control plane is available, kubeadm executes the following actions:Labels the node as control-plane with node-role.kubernetes.io/control-plane=""Taints the node with node-role.kubernetes.io/control-plane:NoSchedulePlease note that the phase to mark the control-plane phase can be invoked individually with the kubeadm init phase mark-control-plane command.Configure TLS-Bootstrapping for node joiningKubeadm uses Authenticating with Bootstrap Tokens for joining new nodes to an existing cluster; for more details see also design proposal.kubeadm init ensures that everything is properly configured for this process, and this includes following steps as well as setting API server and controller flags as already described in previous paragraphs.Note:TLS bootstrapping for nodes can be configured with the command kubeadm init phase bootstrap-token, executing all the configuration steps described in following paragraphs; alternatively, each step can be invoked individually.Create a bootstrap tokenkubeadm init creates a first bootstrap token, either generated automatically or provided by the user with the --token flag; as documented in bootstrap token specification, token should be saved as a secret with name bootstrap-token-<token-id> under kube-system namespace.Please note that:The default token created by kubeadm init will be used to validate temporary user during TLS bootstrap process; those users will be member of system:bootstrappers:kubeadm:default-node-token groupThe token has a limited validity, default 24 hours (the interval may be changed with the —token-ttl flag)Additional tokens can be created with the kubeadm token command, that provide other useful functions for token management as well.Allow joining nodes to call CSR APIKubeadm ensures that users in system:bootstrappers:kubeadm:default-node-token group are able to access the certificate signing API.This is implemented by creating a ClusterRoleBinding named kubeadm:kubelet-bootstrap between the group above and the default RBAC role system:node-bootstrapper.Set up auto approval for new bootstrap tokensKubeadm ensures that the Bootstrap Token will get its CSR request automatically approved by the csrapprover controller.This is implemented by creating ClusterRoleBinding named kubeadm:node-autoapprove-bootstrap between the system:bootstrappers:kubeadm:default-node-token group and the default role system:certificates.k8s.io:certificatesigningrequests:nodeclient.The role system:certificates.k8s.io:certificatesigningrequests:nodeclient should be created as well, granting POST permission to /apis/certificates.k8s.io/certificatesigningrequests/nodeclient.Set up nodes certificate rotation with auto approvalKubeadm ensures that certificate rotation is enabled for nodes, and that a new certificate request for nodes will get its CSR request automatically approved by the csrapprover controller.This is implemented by creating ClusterRoleBinding named kubeadm:node-autoapprove-certificate-rotation between the system:nodes group and the default role system:certificates.k8s.io:certificatesigningrequests:selfnodeclient.Create the public cluster-info ConfigMapThis phase creates the cluster-info ConfigMap in the kube-public namespace.Additionally, it creates a Role and a RoleBinding granting access to the ConfigMap for unauthenticated users (i.e. users in RBAC group system:unauthenticated).Note:The access to the cluster-info ConfigMap is not rate-limited. This may or may not be a problem if you expose your cluster's API server to the internet; worst-case scenario here is a DoS attack where an attacker uses all the in-flight requests the kube-apiserver can handle to serve the cluster-info ConfigMap.Install addonsKubeadm installs the internal DNS server and the kube-proxy addon components via the API server.Note:This phase can be invoked individually with the command kubeadm init phase addon all.proxyA ServiceAccount for kube-proxy is created in the kube-system namespace; then kube-proxy is deployed as a DaemonSet:The credentials (ca.crt and token) to the control plane come from the ServiceAccountThe location (URL) of the API server comes from a ConfigMapThe kube-proxy ServiceAccount is bound to the privileges in the system:node-proxier ClusterRoleDNSThe CoreDNS service is named kube-dns for compatibility reasons with the legacy kube-dns addon.A ServiceAccount for CoreDNS is created in the kube-system namespace.The coredns ServiceAccount is bound to the privileges in the system:coredns ClusterRoleIn Kubernetes version 1.21, support for using kube-dns with kubeadm was removed. You can use CoreDNS with kubeadm even when the related Service is named kube-dns.kubeadm join phases internal designSimilarly to kubeadm init, also kubeadm join internal workflow consists of a sequence of atomic work tasks to perform.This is split into discovery (having the Node trust the Kubernetes API Server) and TLS bootstrap (having the Kubernetes API Server trust the Node).see Authenticating with Bootstrap Tokens or the corresponding design proposal.Preflight checkskubeadm executes a set of preflight checks before starting the join, with the aim to verify preconditions and avoid common cluster startup problems.Also note that:kubeadm join preflight checks are basically a subset of kubeadm init preflight checksIf you are joining a Windows node, Linux specific controls are skipped.In any case the user can skip specific preflight checks (or eventually all preflight checks) with the --ignore-preflight-errors option.Discovery cluster-infoThere are 2 main schemes for discovery. The first is to use a shared token along with the IP address of the API server. The second is to provide a file (that is a subset of the standard kubeconfig file).Shared token discoveryIf kubeadm join is invoked with --discovery-token, token discovery is used; in this case the node basically retrieves the cluster CA certificates from the cluster-info ConfigMap in the kube-public namespace.In order to prevent "man in the middle" attacks, several steps are taken:First, the CA certificate is retrieved via insecure connection (this is possible because kubeadm init is granted access to cluster-info users for system:unauthenticated)Then the CA certificate goes through following validation steps:Basic validation: using the token ID against a JWT signaturePub key validation: using provided --discovery-token-ca-cert-hash. This value is available in the output of kubeadm init or can be calculated using standard tools (the hash is calculated over the bytes of the Subject Public Key Info (SPKI) object as in RFC7469). The --discovery-token-ca-cert-hash flag may be repeated multiple times to allow more than one public key.As an additional validation, the CA certificate is retrieved via secure connection and then compared with the CA retrieved initiallyNote:You can skip CA validation by passing the --discovery-token-unsafe-skip-ca-verification flag on the command line. This weakens the kubeadm security model since others can potentially impersonate the Kubernetes API server.File/https discoveryIf kubeadm join is invoked with --discovery-file, file discovery is used; this file can be a local file or downloaded via an HTTPS URL; in case of HTTPS, the host installed CA bundle is used to verify the connection.With file discovery, the cluster CA certificate is provided into the file itself; in fact, the discovery file is a kubeconfig file with only server and certificate-authority-data attributes set, as described in the kubeadm join reference doc; when the connection with the cluster is established, kubeadm tries to access the cluster-info ConfigMap, and if available, uses it.TLS BootstrapOnce the cluster info is known, the file bootstrap-kubelet.conf is written, thus allowing kubelet to do TLS Bootstrapping.The TLS bootstrap mechanism uses the shared token to temporarily authenticate with the Kubernetes API server to submit a certificate signing request (CSR) for a locally created key pair.The request is then automatically approved and the operation completes saving ca.crt file and kubelet.conf file to be used by the kubelet for joining the cluster, while bootstrap-kubelet.conf is deleted.Note:The temporary authentication is validated against the token saved during the kubeadm init process (or with additional tokens created with kubeadm token command)The temporary authentication resolves to a user member of system:bootstrappers:kubeadm:default-node-token group which was granted access to the CSR api during the kubeadm init processThe automatic CSR approval is managed by the csrapprover controller, according to the configuration present in the kubeadm init processkubeadm upgrade workflow internal designkubeadm upgrade has sub-commands for handling the upgrade of the Kubernetes cluster created by kubeadm. You must run kubeadm upgrade apply on a control plane node (you can choose which one); this starts the upgrade process. You then run kubeadm upgrade node on all remaining nodes (both worker nodes and control plane nodes).Both kubeadm upgrade apply and kubeadm upgrade node have a phase subcommand which provides access to the internal phases of the upgrade process. See kubeadm upgrade phase for more details.Additional utility upgrade commands are kubeadm upgrade plan and kubeadm upgrade diff.All upgrade sub-commands support passing a configuration file.kubeadm upgrade planYou can optionally run kubeadm upgrade plan before you run kubeadm upgrade apply. The plan subcommand checks which versions are available to upgrade to and validates whether your current cluster is upgradeable.kubeadm upgrade diffThis shows what differences would be applied to existing static pod manifests for control plane nodes. A more verbose way to do the same thing is running kubeadm upgrade apply --dry-run or kubeadm upgrade node --dry-run.kubeadm upgrade applykubeadm upgrade apply prepares the cluster for the upgrade of all nodes, and also upgrades the control plane node where it's run. The steps it performs are:Runs preflight checks similarly to kubeadm init and kubeadm join, ensuring container images are downloaded and the cluster is in a good state to be upgraded.Upgrades the control plane manifest files on disk in /etc/kubernetes/manifests and waits for the kubelet to restart the components if the files have changed.Uploads the updated kubeadm and kubelet configurations to the cluster in the kubeadm-config and the kubelet-config ConfigMaps (both in the kube-system namespace).Writes updated kubelet configuration for this node in /var/lib/kubelet/config.yaml, and read the node's /var/lib/kubelet/instance-config.yaml file and patch fields like containerRuntimeEndpoint from this instance configuration into /var/lib/kubelet/config.yaml.Configures bootstrap token and the cluster-info ConfigMap for RBAC rules. This is the same as in the kubeadm init stage and ensures that the cluster continues to support nodes joining with bootstrap tokens.Upgrades the kube-proxy and CoreDNS addons conditionally if all existing kube-apiservers in the cluster have already been upgraded to the target version.Performs any post-upgrade tasks, such as, cleaning up deprecated features which are release specific.kubeadm upgrade nodekubeadm upgrade node upgrades a single control plane or worker node after the cluster upgrade has started (by running kubeadm upgrade apply). The command detects if the node is a control plane node by checking if the file /etc/kubernetes/manifests/kube-apiserver.yaml exists. On finding that file, the kubeadm tool infers that there is a running kube-apiserver Pod on this node.Runs preflight checks similarly to kubeadm upgrade apply.For control plane nodes, upgrades the control plane manifest files on disk in /etc/kubernetes/manifests and waits for the kubelet to restart the components if the files have changed.Writes updated kubelet configuration for this node in /var/lib/kubelet/config.yaml, and read the node's /var/lib/kubelet/instance-config.yaml file and patch fields like containerRuntimeEndpoint from this instance configuration into /var/lib/kubelet/config.yaml.(For control plane nodes) upgrades the kube-proxy and CoreDNS addons conditionally, provided that all existing API servers in the cluster have already been upgraded to the target version.Performs any post-upgrade tasks, such as cleaning up deprecated features which are release specific.kubeadm reset workflow internal designYou can use the kubeadm reset subcommand on a node where kubeadm commands previously executed. This subcommand performs a best-effort cleanup of the node. If certain actions fail you must intervene and perform manual cleanup.The command supports phases. See kubeadm reset phase for more details.The command supports a configuration file.Additionally:IPVS, iptables and nftables rules are not cleaned up.CNI (network plugin) configuration is not cleaned up..kube/ in the user's home directory is not cleaned up.The command has the following stages:Runs preflight checks on the node to determine if its healthy.For control plane nodes, removes any local etcd member data.Stops the kubelet.Stops running containers.Unmounts any mounted directories in /var/lib/kubelet.Deletes any files and directories managed by kubeadm in /var/lib/kubelet and /etc/kubernetes.