# Volume Health Monitoring

Volume Health MonitoringFEATURE STATE: Kubernetes v1.21 [alpha]CSI volume health monitoring allows CSI Drivers to detect abnormal volume conditions from the underlying storage systems and report them as events on PVCs or Pods.Volume health monitoringKubernetes volume health monitoring is part of how Kubernetes implements the Container Storage Interface (CSI). Volume health monitoring feature is implemented in two components: an External Health Monitor controller, and the kubelet.If a CSI Driver supports Volume Health Monitoring feature from the controller side, an event will be reported on the related PersistentVolumeClaim (PVC) when an abnormal volume condition is detected on a CSI volume.The External Health Monitor controller also watches for node failure events. You can enable node failure monitoring by setting the enable-node-watcher flag to true. When the external health monitor detects a node failure event, the controller reports an Event will be reported on the PVC to indicate that pods using this PVC are on a failed node.If a CSI Driver supports Volume Health Monitoring feature from the node side, an Event will be reported on every Pod using the PVC when an abnormal volume condition is detected on a CSI volume. In addition, Volume Health information is exposed as Kubelet VolumeStats metrics. A new metric kubelet_volume_stats_health_status_abnormal is added. This metric includes two labels: namespace and persistentvolumeclaim. The count is either 1 or 0. 1 indicates the volume is unhealthy, 0 indicates volume is healthy. For more information, please check KEP.Note:You need to enable the CSIVolumeHealth feature gate to use this feature from the node side.What's nextSee the CSI driver documentation to find out which CSI drivers have implemented this feature.


# Manage TLS Certificates in a Cluster

Manage TLS Certificates in a ClusterKubernetes provides a certificates.k8s.io API, which lets you provision TLS certificates signed by a Certificate Authority (CA) that you control. These CA and certificates can be used by your workloads to establish trust.certificates.k8s.io API uses a protocol that is similar to the ACME draft.Note:Certificates created using the certificates.k8s.io API are signed by a dedicated CA. It is possible to configure your cluster to use the cluster root CA for this purpose, but you should never rely on this. Do not assume that these certificates will validate against the cluster root CA.Before you beginYou need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. It is recommended to run this tutorial on a cluster with at least two nodes that are not acting as control plane hosts. If you do not already have a cluster, you can create one by using minikube or you can use one of these Kubernetes playgrounds:iximiuz LabsKillercodaKodeKloudPlay with KubernetesYou need the cfssl tool. You can download cfssl from https://github.com/cloudflare/cfssl/releases.Some steps in this page use the jq tool. If you don't have jq, you can install it via your operating system's software sources, or fetch it from https://jqlang.github.io/jq/.Trusting TLS in a clusterTrusting the custom CA from an application running as a pod usually requires some extra application configuration. You will need to add the CA certificate bundle to the list of CA certificates that the TLS client or server trusts. For example, you would do this with a golang TLS config by parsing the certificate chain and adding the parsed certificates to the RootCAs field in the tls.Config struct.Note:Even though the custom CA certificate may be included in the filesystem (in the ConfigMap kube-root-ca.crt), you should not use that certificate authority for any purpose other than to verify internal Kubernetes endpoints. An example of an internal Kubernetes endpoint is the Service named kubernetes in the default namespace.If you want to use a custom certificate authority for your workloads, you should generate that CA separately, and distribute its CA certificate using a ConfigMap that your pods have access to read.Requesting a certificateThe following section demonstrates how to create a TLS certificate for a Kubernetes service accessed through DNS.Note:This tutorial uses CFSSL: Cloudflare's PKI and TLS toolkit click here to know more.Create a certificate signing requestGenerate a private key and certificate signing request (or CSR) by running the following command:cat <<EOF | cfssl genkey - | cfssljson -bare server { "hosts": [ "my-svc.my-namespace.svc.cluster.local", "my-pod.my-namespace.pod.cluster.local", "192.0.2.24", "10.0.34.2" ], "CN": "my-pod.my-namespace.pod.cluster.local", "key": { "algo": "ecdsa", "size": 256 } } EOF Where 192.0.2.24 is the service's cluster IP, my-svc.my-namespace.svc.cluster.local is the service's DNS name, 10.0.34.2 is the pod's IP and my-pod.my-namespace.pod.cluster.local is the pod's DNS name. You should see the output similar to:2022/02/01 11:45:32 [INFO] generate received request 2022/02/01 11:45:32 [INFO] received CSR 2022/02/01 11:45:32 [INFO] generating key: ecdsa-256 2022/02/01 11:45:32 [INFO] encoded CSR This command generates two files; it generates server.csr containing the PEM encoded PKCS#10 certification request, and server-key.pem containing the PEM encoded key to the certificate that is still to be created.Create a CertificateSigningRequest object to send to the Kubernetes APIGenerate a CSR manifest (in YAML), and send it to the API server. You can do that by running the following command:cat <<EOF | kubectl apply -f - apiVersion: certificates.k8s.io/v1 kind: CertificateSigningRequest metadata: name: my-svc.my-namespace spec: request: $(cat server.csr | base64 | tr -d '\n') signerName: example.com/serving usages: - digital signature - key encipherment - server auth EOF Notice that the server.csr file created in step 1 is base64 encoded and stashed in the .spec.request field. You are also requesting a certificate with the "digital signature", "key encipherment", and "server auth" key usages, signed by an example example.com/serving signer. A specific signerName must be requested. View documentation for supported signer names for more information.The CSR should now be visible from the API in a Pending state. You can see it by running:kubectl describe csr my-svc.my-namespace Name: my-svc.my-namespace Labels: <none> Annotations: <none> CreationTimestamp: Tue, 01 Feb 2022 11:49:15 -0500 Requesting User: yourname@example.com Signer: example.com/serving Status: Pending Subject: Common Name: my-pod.my-namespace.pod.cluster.local Serial Number: Subject Alternative Names: DNS Names: my-pod.my-namespace.pod.cluster.local my-svc.my-namespace.svc.cluster.local IP Addresses: 192.0.2.24 10.0.34.2 Events: <none> Get the CertificateSigningRequest approvedApproving the certificate signing request is either done by an automated approval process or on a one off basis by a cluster administrator. If you're authorized to approve a certificate request, you can do that manually using kubectl; for example:kubectl certificate approve my-svc.my-namespace certificatesigningrequest.certificates.k8s.io/my-svc.my-namespace approved You should now see the following:kubectl get csr NAME AGE SIGNERNAME REQUESTOR REQUESTEDDURATION CONDITION my-svc.my-namespace 10m example.com/serving yourname@example.com <none> Approved This means the certificate request has been approved and is waiting for the requested signer to sign it.Sign the CertificateSigningRequestNext, you'll play the part of a certificate signer, issue the certificate, and upload it to the API.A signer would typically watch the CertificateSigningRequest API for objects with its signerName, check that they have been approved, sign certificates for those requests, and update the API object status with the issued certificate.Create a Certificate AuthorityYou need an authority to provide the digital signature on the new certificate.First, create a signing certificate by running the following:cat <<EOF | cfssl gencert -initca - | cfssljson -bare ca { "CN": "My Example Signer", "key": { "algo": "rsa", "size": 2048 } } EOF You should see output similar to:2022/02/01 11:50:39 [INFO] generating a new CA key and certificate from CSR 2022/02/01 11:50:39 [INFO] generate received request 2022/02/01 11:50:39 [INFO] received CSR 2022/02/01 11:50:39 [INFO] generating key: rsa-2048 2022/02/01 11:50:39 [INFO] encoded CSR 2022/02/01 11:50:39 [INFO] signed certificate with serial number 263983151013686720899716354349605500797834580472 This produces a certificate authority key file (ca-key.pem) and certificate (ca.pem).Issue a certificatetls/server-signing-config.json { "signing": { "default": { "usages": [ "digital signature", "key encipherment", "server auth" ], "expiry": "876000h", "ca_constraint": { "is_ca": false } } } }Use a server-signing-config.json signing configuration and the certificate authority key file and certificate to sign the certificate request:kubectl get csr my-svc.my-namespace -o jsonpath='{.spec.request}' | \ base64 --decode | \ cfssl sign -ca ca.pem -ca-key ca-key.pem -config server-signing-config.json - | \ cfssljson -bare ca-signed-server You should see the output similar to:2022/02/01 11:52:26 [INFO] signed certificate with serial number 576048928624926584381415936700914530534472870337 This produces a signed serving certificate file, ca-signed-server.pem.Upload the signed certificateFinally, populate the signed certificate in the API object's status:kubectl get csr my-svc.my-namespace -o json | \ jq '.status.certificate = "'$(base64 ca-signed-server.pem | tr -d '\n')'"' | \ kubectl replace --raw /apis/certificates.k8s.io/v1/certificatesigningrequests/my-svc.my-namespace/status -f - Note:This uses the command line tool jq to populate the base64-encoded content in the .status.certificate field. If you do not have jq, you can also save the JSON output to a file, populate this field manually, and upload the resulting file.Once the CSR is approved and the signed certificate is uploaded, run:kubectl get csr The output is similar to:NAME AGE SIGNERNAME REQUESTOR REQUESTEDDURATION CONDITION my-svc.my-namespace 20m example.com/serving yourname@example.com <none> Approved,Issued Download the certificate and use itNow, as the requesting user, you can download the issued certificate and save it to a server.crt file by running the following:kubectl get csr my-svc.my-namespace -o jsonpath='{.status.certificate}' \ | base64 --decode > server.crt Now you can populate server.crt and server-key.pem in a Secret that you could later mount into a Pod (for example, to use with a webserver that serves HTTPS).kubectl create secret tls server --cert server.crt --key server-key.pem secret/server created Finally, you can populate ca.pem into a ConfigMap and use it as the trust root to verify the serving certificate:kubectl create configmap example-serving-ca --from-file ca.crt=ca.pem configmap/example-serving-ca created Approving CertificateSigningRequestsA Kubernetes administrator (with appropriate permissions) can manually approve (or deny) CertificateSigningRequests by using the kubectl certificate approve and kubectl certificate deny commands. However if you intend to make heavy usage of this API, you might consider writing an automated certificates controller.Caution:The ability to approve CSRs decides who trusts whom within your environment. The ability to approve CSRs should not be granted broadly or lightly.You should make sure that you confidently understand both the verification requirements that fall on the approver and the repercussions of issuing a specific certificate before you grant the approve permission.Whether a machine or a human using kubectl as above, the role of the approver is to verify that the CSR satisfies two requirements:The subject of the CSR controls the private key used to sign the CSR. This addresses the threat of a third party masquerading as an authorized subject. In the above example, this step would be to verify that the pod controls the private key used to generate the CSR.The subject of the CSR is authorized to act in the requested context. This addresses the threat of an undesired subject joining the cluster. In the above example, this step would be to verify that the pod is allowed to participate in the requested service.If and only if these two requirements are met, the approver should approve the CSR and otherwise should deny the CSR.For more information on certificate approval and access control, read the Certificate Signing Requests reference page.Configuring your cluster to provide signingThis page assumes that a signer is set up to serve the certificates API. The Kubernetes controller manager provides a default implementation of a signer. To enable it, pass the --cluster-signing-cert-file and --cluster-signing-key-file parameters to the controller manager with paths to your Certificate Authority's keypair.


# CronJob

CronJobCronJob represents the configuration of a single cron job.apiVersion: batch/v1import "k8s.io/api/batch/v1"CronJobCronJob represents the configuration of a single cron job.apiVersion: batch/v1kind: CronJobmetadata (ObjectMeta)Standard object's metadata. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadataspec (CronJobSpec)Specification of the desired behavior of a cron job, including the schedule. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-statusstatus (CronJobStatus)Current status of a cron job. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-statusCronJobSpecCronJobSpec describes how the job execution will look like and when it will actually run.jobTemplate (JobTemplateSpec), requiredSpecifies the job that will be created when executing a CronJob.JobTemplateSpec describes the data a Job should have when created from a templatejobTemplate.metadata (ObjectMeta)Standard object's metadata of the jobs created from this template. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadatajobTemplate.spec (JobSpec)Specification of the desired behavior of the job. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-statusschedule (string), requiredThe schedule in Cron format, see https://en.wikipedia.org/wiki/Cron.timeZone (string)The time zone name for the given schedule, see https://en.wikipedia.org/wiki/List_of_tz_database_time_zones. If not specified, this will default to the time zone of the kube-controller-manager process. The set of valid time zone names and the time zone offset is loaded from the system-wide time zone database by the API server during CronJob validation and the controller manager during execution. If no system-wide time zone database can be found a bundled version of the database is used instead. If the time zone name becomes invalid during the lifetime of a CronJob or due to a change in host configuration, the controller will stop creating new new Jobs and will create a system event with the reason UnknownTimeZone. More information can be found in https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/#time-zonesconcurrencyPolicy (string)Specifies how to treat concurrent executions of a Job. Valid values are:"Allow" (default): allows CronJobs to run concurrently; - "Forbid": forbids concurrent runs, skipping


# SelfSubjectReview

SelfSubjectReviewSelfSubjectReview contains the user information that the kube-apiserver has about the user making this request.apiVersion: authentication.k8s.io/v1import "k8s.io/api/authentication/v1"SelfSubjectReviewSelfSubjectReview contains the user information that the kube-apiserver has about the user making this request. When using impersonation, users will receive the user info of the user being impersonated. If impersonation or request header authentication is used, any extra keys will have their case ignored and returned as lowercase.apiVersion: authentication.k8s.io/v1kind: SelfSubjectReviewmetadata (ObjectMeta)Standard object's metadata. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadatastatus (SelfSubjectReviewStatus)Status is filled in by the server with the user attributes.SelfSubjectReviewStatusSelfSubjectReviewStatus is filled by the kube-apiserver and sent back to a user.userInfo (UserInfo)User attributes of the user making this request.UserInfo holds the information about the user needed to implement the user.Info interface.userInfo.extra (map[string][]string)Any additional information provided by the authenticator.userInfo.groups ([]string)Atomic: will be replaced during a mergeThe names of groups this user is a part of.userInfo.uid (string)A unique value that identifies this user across time. If this user is deleted and another user by the same name is added, they will have different UIDs.userInfo.username (string)The name that uniquely identifies this user among all active users.Operationscreate create a SelfSubjectReviewHTTP RequestPOST /apis/authentication.k8s.io/v1/selfsubjectreviewsParametersbody: SelfSubjectReview, requireddryRun (in query): stringdryRunfieldManager (in query): stringfieldManagerfieldValidation (in query): stringfieldValidationpretty (in query): stringprettyResponse200 (SelfSubjectReview): OK201 (SelfSubjectReview): Created202 (SelfSubjectReview): Accepted401: UnauthorizedThis page is automatically generated.If you plan to report an issue with this page, mention that the page is auto-generated in your issue description. The fix may need to happen elsewhere in the Kubernetes project.


# Feature Gates (removed)

Feature Gates (removed)This page contains list of feature gates that have been removed. The information on this page is for reference. A removed feature gate is different from a GA'ed or deprecated one in that a removed one is no longer recognized as a valid feature gate. However, a GA'ed or a deprecated feature gate is still recognized by the corresponding Kubernetes components although they are unable to cause any behavior differences in a cluster.For feature gates that are still recognized by the Kubernetes components, please refer to the Alpha/Beta feature gate table or the Graduated/Deprecated feature gate tableFeature gates that are removedIn the following table:The "From" column contains the Kubernetes release when a feature is introduced or its release stage is changed.The "To" column, if not empty, contains the last Kubernetes release in which you can still use a feature gate. If the feature stage is either "Deprecated" or "GA", the "To" column is the Kubernetes release when the feature is removed.Feature Gates RemovedFeatureDefaultStageFromToAcceleratorsfalseAlpha1.61.10Accelerators–Deprecated1.111.11AdmissionWebhookMatchConditionsfalseAlpha1.271.27AdmissionWebhookMatchConditionstrueBeta1.281.29AdmissionWebhookMatchConditionstrueGA1.301.32AdvancedAuditingfalseAlpha1.71.7AdvancedAuditingtrueBeta1.81.11AdvancedAuditingtrueGA1.121.27AffinityInAnnotationsfalseAlpha1.61.7AffinityInAnnotations–Deprecated1.81.8AggregatedDiscoveryEndpointfalseAlpha1.261.26AggregatedDiscoveryEndpointtrueBeta1.271.29AggregatedDiscoveryEndpointtrueGA1.301.32AllowExtTrafficLocalEndpointsfalseBeta1.41.6AllowExtTrafficLocalEndpointstrueGA1.71.9AllowInsecureBackendProxytrueBeta1.171.20AllowInsecureBackendProxytrueGA1.211.25APIListChunkingfalseAlpha1.81.8APIListChunkingtrueBeta1.91.28APIListChunkingtrueGA1.291.32APIPriorityAndFairnessfalseAlpha1.181.19APIPriorityAndFairnesstrueBeta1.201.28APIPriorityAndFairnesstrueGA1.291.30APISelfSubjectReviewfalseAlpha1.261.26APISelfSubjectReviewtrueBeta1.271.27APISelfSubjectReviewtrueGA1.281.29AppArmortrueBeta1.41.30AppArmortrueGA1.311.32AppArmorFieldstrueBeta1.301.30AppArmorFieldstrueGA1.311.32AttachVolumeLimitfalseAlpha1.111.11AttachVolumeLimittrueBeta1.121.16AttachVolumeLimittrueGA1.171.21BalanceAttachedNodeVolumesfalseAlpha1.111.21BalanceAttachedNodeVolumesfalseDeprecated1.221.22BlockVolumefalseAlpha1.91.12BlockVolumetrueBeta1.131.17BlockVolumetrueGA1.181.21BoundServiceAccountTokenVolumefalseAlpha1.131.20BoundServiceAccountTokenVolumetrueBeta1.211.21BoundServiceAccountTokenVolumetrueGA1.221.23CloudDualStackNodeIPsfalseAlpha1.271.28CloudDualStackNodeIPstrueBeta1.291.29CloudDualStackNodeIPstrueGA1.301.31ConfigurableFSGroupPolicyfalseAlpha1.181.19ConfigurableFSGroupPolicytrueBeta1.201.22ConfigurableFSGroupPolicytrueGA1.231.25ConsistentHTTPGetHandlerstrueGA1.251.30ControllerManagerLeaderMigrationfalseAlpha1.211.21ControllerManagerLeaderMigrationtrueBeta1.221.23ControllerManagerLeaderMigrationtrueGA1.241.26CPUManagerfalseAlpha1.81.9CPUManagertrueBeta1.101.25CPUManagertrueGA1.261.32CRIContainerLogRotationfalseAlpha1.101.10CRIContainerLogRotationtrueBeta1.111.20CRIContainerLogRotationtrueGA1.211.22CronJobControllerV2falseAlpha1.201.20CronJobControllerV2trueBeta1.211.21CronJobControllerV2trueGA1.221.23CronJobTimeZonefalseAlpha1.241.24CronJobTimeZonetrueBeta1.251.26CronJobTimeZonetrueGA1.271.28CSIBlockVolumefalseAlpha1.111.13CSIBlockVolumetrueBeta1.141.17CSIBlockVolumetrueGA1.181.21CSIDriverRegistryfalseAlpha1.121.13CSIDriverRegistrytrueBeta1.141.17CSIDriverRegistrytrueGA1.181.21CSIInlineVolumefalseAlpha1.151.15CSIInlineVolumetrueBeta1.161.24CSIInlineVolumetrueGA1.251.26CSIMigrationfalseAlpha1.141.16CSIMigrationtrueBeta1.171.24CSIMigrationtrueGA1.251.26CSIMigrationAWSfalseAlpha1.141.16CSIMigrationAWSfalseBeta1.171.22CSIMigrationAWStrueBeta1.231.24CSIMigrationAWStrueGA1.251.26CSIMigrationAWSCompletefalseAlpha1.171.20CSIMigrationAWSComplete–Deprecated1.211.21CSIMigrationAzureDiskfalseAlpha1.151.18CSIMigrationAzureDiskfalseBeta1.191.22CSIMigrationAzureDisktrueBeta1.231.23CSIMigrationAzureDisktrueGA1.241.26CSIMigrationAzureDiskCompletefalseAlpha1.171.20CSIMigrationAzureDiskComplete–Deprecated1.211.21CSIMigrationAzureFilefalseAlpha1.151.20CSIMigrationAzureFilefalseBeta1.211.23CSIMigrationAzureFiletrueBeta1.241.25CSIMigrationAzureFiletrueGA1.261.29CSIMigrationAzureFileCompletefalseAlpha1.171.20CSIMigrationAzureFileComplete–Deprecated1.211.21CSIMigrationGCEfalseAlpha1.141.16CSIMigrationGCEfalseBeta1.171.22CSIMigrationGCEtrueBeta1.231.24CSIMigrationGCEtrueGA1.251.27CSIMigrationGCECompletefalseAlpha1.171.20CSIMigrationGCEComplete–Deprecated1.211.21CSIMigrationOpenStackfalseAlpha1.141.17CSIMigrationOpenStacktrueBeta1.181.23CSIMigrationOpenStacktrueGA1.241.25CSIMigrationOpenStackCompletefalseAlpha1.171.20CSIMigrationOpenStackComplete–Deprecated1.211.21CSIMigrationRBDfalseAlpha1.231.27CSIMigrationRBDfalseDeprecated1.281.30CSIMigrationvSpherefalseAlpha1.181.18CSIMigrationvSpherefalseBeta1.191.24CSIMigrationvSpheretrueBeta1.251.25CSIMigrationvSpheretrueGA1.261.28CSIMigrationvSphereCompletefalseBeta1.191.21CSIMigrationvSphereComplete–Deprecated1.221.22CSINodeExpandSecretfalseAlpha1.251.26CSINodeExpandSecrettrueBeta1.271.28CSINodeExpandSecrettrueGA1.291.30CSINodeInfofalseAlpha1.121.13CSINodeInfotrueBeta1.141.16CSINodeInfotrueGA1.171.22CSIPersistentVolumefalseAlpha1.91.9CSIPersistentVolumetrueBeta1.101.12CSIPersistentVolumetrueGA1.131.16CSIServiceAccountTokenfalseAlpha1.201.20CSIServiceAccountTokentrueBeta1.211.21CSIServiceAccountTokentrueGA1.221.24CSIStorageCapacityfalseAlpha1.191.20CSIStorageCapacitytrueBeta1.211.23CSIStorageCapacitytrueGA1.241.27CSIVolumeFSGroupPolicyfalseAlpha1.191.19CSIVolumeFSGroupPolicytrueBeta1.201.22CSIVolumeFSGroupPolicytrueGA1.231.25CSRDurationtrueBeta1.221.23CSRDurationtrueGA1.241.25CustomPodDNSfalseAlpha1.91.9CustomPodDNStrueBeta1.101.13CustomPodDNStrueGA1.141.16CustomResourceDefaultingfalseAlpha1.151.15CustomResourceDefaultingtrueBeta1.161.16CustomResourceDefaultingtrueGA1.171.18CustomResourcePublishOpenAPIfalseAlpha1.141.14CustomResourcePublishOpenAPItrueBeta1.151.15CustomResourcePublishOpenAPItrueGA1.161.18CustomResourceSubresourcesfalseAlpha1.101.10CustomResourceSubresourcestrueBeta1.111.15CustomResourceSubresourcestrueGA1.161.18CustomResourceValidationfalseAlpha1.81.8CustomResourceValidationtrueBeta1.91.15CustomResourceValidationtrueGA1.161.18CustomResourceValidationExpressionsfalseAlpha1.231.24CustomResourceValidationExpressionstrueBeta1.251.28CustomResourceValidationExpressionstrueGA1.291.30CustomResourceWebhookConversionfalseAlpha1.131.14CustomResourceWebhookConversiontrueBeta1.151.15CustomResourceWebhookConversiontrueGA1.161.18DaemonSetUpdateSurgefalseAlpha1.211.21DaemonSetUpdateSurgetrueBeta1.221.24DaemonSetUpdateSurgetrueGA1.251.26DefaultHostNetworkHostPortsInPodTemplatesfalseDeprecated1.281.30DefaultPodTopologySpreadfalseAlpha1.191.19DefaultPodTopologySpreadtrueBeta1.201.23DefaultPodTopologySpreadtrueGA1.241.25DelegateFSGroupToCSIDriverfalseAlpha1.221.22DelegateFSGroupToCSIDrivertrueBeta1.231.25DelegateFSGroupToCSIDrivertrueGA1.261.27DevicePluginCDIDevicesfalseAlpha1.281.28DevicePluginCDIDevicestrueBeta1.291.30DevicePluginCDIDevicestrueGA1.311.33DevicePluginsfalseAlpha1.81.9DevicePluginstrueBeta1.101.25DevicePluginstrueGA1.261.27DisableAcceleratorUsageMetricsfalseAlpha1.191.19DisableAcceleratorUsageMetricstrueBeta1.201.24DisableAcceleratorUsageMetricstrueGA1.251.27DisableCloudProvidersfalseAlpha1.221.28DisableCloudProviderstrueBeta1.291.30DisableCloudProviderstrueGA1.311.32DisableKubeletCloudCredentialProvidersfalseAlpha1.231.28DisableKubeletCloudCredentialProviderstrueBeta1.291.30DisableKubeletCloudCredentialProviderstrueGA1.311.32DownwardAPIHugePagesfalseAlpha1.201.20DownwardAPIHugePagesfalseBeta1.211.21DownwardAPIHugePagestrueBeta1.221.26DownwardAPIHugePagestrueGA1.271.28DRAControlPlaneControllerfalseAlpha1.261.31DryRunfalseAlpha1.121.12DryRuntrueBeta1.131.18DryRuntrueGA1.191.27DynamicAuditingfalseAlpha1.131.18DynamicAuditing–Deprecated1.191.19DynamicKubeletConfigfalseAlpha1.41.10DynamicKubeletConfigtrueBeta1.111.21DynamicKubeletConfigfalseDeprecated1.221.25DynamicProvisioningSchedulingfalseAlpha1.111.11DynamicProvisioningScheduling–Deprecated1.12–DynamicVolumeProvisioningtrueAlpha1.31.7DynamicVolumeProvisioningtrueGA1.81.12EfficientWatchResumptionfalseAlpha1.201.20EfficientWatchResumptiontrueBeta1.211.23EfficientWatchResumptiontrueGA1.241.32EnableAggregatedDiscoveryTimeouttrueDeprecated1.161.17EnableEquivalenceClassCachefalseAlpha1.81.12EnableEquivalenceClassCache–Deprecated1.131.23EndpointSlicefalseAlpha1.161.16EndpointSlicefalseBeta1.171.17EndpointSlicetrueBeta1.181.20EndpointSlicetrueGA1.211.24EndpointSliceNodeNamefalseAlpha1.201.20EndpointSliceNodeNametrueGA1.211.24EndpointSliceProxyingfalseAlpha1.181.18EndpointSliceProxyingtrueBeta1.191.21EndpointSliceProxyingtrueGA1.221.24EndpointSliceTerminatingConditionfalseAlpha1.201.21EndpointSliceTerminatingConditiontrueBeta1.221.25EndpointSliceTerminatingConditiontrueGA1.261.27EphemeralContainersfalseAlpha1.161.22EphemeralContainerstrueBeta1.231.24EphemeralContainerstrueGA1.251.26EvenPodsSpreadfalseAlpha1.161.17EvenPodsSpreadtrueBeta1.181.18EvenPodsSpreadtrueGA1.191.21ExpandCSIVolumesfalseAlpha1.141.15ExpandCSIVolumestrueBeta1.161.23ExpandCSIVolumestrueGA1.241.26ExpandedDNSConfigfalseAlpha1.221.25ExpandedDNSConfigtrueBeta1.261.27ExpandedDNSConfigtrueGA1.281.29ExpandInUsePersistentVolumesfalseAlpha1.111.14ExpandInUsePersistentVolumestrueBeta1.151.23ExpandInUsePersistentVolumestrueGA1.241.26ExpandPersistentVolumesfalseAlpha1.81.10ExpandPersistentVolumestrueBeta1.111.23ExpandPersistentVolumestrueGA1.241.26ExperimentalCriticalPodAnnotationfalseAlpha1.51.12ExperimentalCriticalPodAnnotationfalseDeprecated1.131.16ExperimentalHostUserNamespaceDefaultingfalseBeta1.51.27ExperimentalHostUserNamespaceDefaultingfalseDeprecated1.281.29ExternalPolicyForExternalIPtrueGA1.181.22GCERegionalPersistentDisktrueBeta1.101.12GCERegionalPersistentDisktrueGA1.131.16GenericEphemeralVolumefalseAlpha1.191.20GenericEphemeralVolumetrueBeta1.211.22GenericEphemeralVolumetrueGA1.231.24GRPCContainerProbefalseAlpha1.231.23GRPCContainerProbetrueBeta1.241.26GRPCContainerProbetrueGA1.271.28HPAContainerMetricsfalseAlpha1.201.26HPAContainerMetricstrueBeta1.271.29HPAContainerMetricstrueGA1.301.31HugePagesfalseAlpha1.81.9HugePagestrueBeta1.101.13HugePagestrueGA1.141.16HugePageStorageMediumSizefalseAlpha1.181.18HugePageStorageMediumSizetrueBeta1.191.21HugePageStorageMediumSizetrueGA1.221.24HyperVContainerfalseAlpha1.101.19HyperVContainerfalseDeprecated1.201.20IdentifyPodOSfalseAlpha1.231.23IdentifyPodOStrueBeta1.241.24IdentifyPodOStrueGA1.251.26ImmutableEphemeralVolumesfalseAlpha1.181.18ImmutableEphemeralVolumestrueBeta1.191.20ImmutableEphemeralVolumestrueGA1.211.24IndexedJobfalseAlpha1.211.21IndexedJobtrueBeta1.221.23IndexedJobtrueGA1.241.25IngressClassNamespacedParamsfalseAlpha1.211.21IngressClassNamespacedParamstrueBeta1.221.22IngressClassNamespacedParamstrueGA1.231.24InitializersfalseAlpha1.71.13Initializers–Deprecated1.141.14InTreePluginAWSUnregisterfalseAlpha1.211.30InTreePluginAzureDiskUnregisterfalseAlpha1.211.30InTreePluginAzureFileUnregisterfalseAlpha1.211.30InTreePluginGCEUnregisterfalseAlpha1.211.30InTreePluginOpenStackUnregisterfalseAlpha1.211.30InTreePluginRBDUnregisterfalseAlpha1.231.27InTreePluginRBDUnregisterfalseDeprecated1.281.30InTreePluginvSphereUnregisterfalseAlpha1.211.30IPTablesOwnershipCleanupfalseAlpha1.251.26IPTablesOwnershipCleanuptrueBeta1.271.27IPTablesOwnershipCleanuptrueGA1.281.29IPv6DualStackfalseAlpha1.151.20IPv6DualStacktrueBeta1.211.22IPv6DualStacktrueGA1.231.24JobMutableNodeSchedulingDirectivestrueBeta1.231.26JobMutableNodeSchedulingDirectivestrueGA1.271.28JobPodFailurePolicyfalseAlpha1.251.25JobPodFailurePolicytrueBeta1.261.30JobPodFailurePolicytrueGA1.311.32JobReadyPodsfalseAlpha1.231.23JobReadyPodstrueBeta1.241.28JobReadyPodstrueGA1.291.30JobTrackingWithFinalizersfalseAlpha1.221.22JobTrackingWithFinalizersfalseBeta1.231.24JobTrackingWithFinalizerstrueBeta1.251.25JobTrackingWithFinalizerstrueGA1.261.28KMSv2falseAlpha1.251.26KMSv2trueBeta1.271.28KMSv2trueGA1.291.31KMSv2KDFfalseBeta1.281.28KMSv2KDFtrueGA1.291.31KubeletConfigFilefalseAlpha1.81.9KubeletConfigFile–Deprecated1.101.10KubeletCredentialProvidersfalseAlpha1.201.23KubeletCredentialProviderstrueBeta1.241.25KubeletCredentialProviderstrueGA1.261.28KubeletPluginsWatcherfalseAlpha1.111.11KubeletPluginsWatchertrueBeta1.121.12KubeletPluginsWatchertrueGA1.131.16KubeletPodResourcesfalseAlpha1.131.14KubeletPodResourcestrueBeta1.151.27KubeletPodResourcestrueGA1.281.29KubeletPodResourcesGetAllocatablefalseAlpha1.211.22KubeletPodResourcesGetAllocatabletrueBeta1.231.27KubeletPodResourcesGetAllocatabletrueGA1.281.29KubeProxyDrainingTerminatingNodesfalseAlpha1.281.30KubeProxyDrainingTerminatingNodestrueBeta1.301.30KubeProxyDrainingTerminatingNodestrueGA1.311.32LegacyNodeRoleBehaviorfalseAlpha1.161.18LegacyNodeRoleBehaviortrueBeta1.191.20LegacyNodeRoleBehaviorfalseGA1.211.22LegacyServiceAccountTokenCleanUpfalseAlpha1.281.28LegacyServiceAccountTokenCleanUptrueBeta1.291.29LegacyServiceAccountTokenCleanUptrueGA1.301.31LegacyServiceAccountTokenNoAutoGenerationtrueBeta1.241.25LegacyServiceAccountTokenNoAutoGenerationtrueGA1.261.28LegacyServiceAccountTokenTrackingfalseAlpha1.261.26LegacyServiceAccountTokenTrackingtrueBeta1.271.27LegacyServiceAccountTokenTrackingtrueGA1.281.29LocalStorageCapacityIsolationfalseAlpha1.71.9LocalStorageCapacityIsolationtrueBeta1.101.24LocalStorageCapacityIsolationtrueGA1.251.26MinDomainsInPodTopologySpreadfalseAlpha1.241.24MinDomainsInPodTopologySpreadfalseBeta1.251.26MinDomainsInPodTopologySpreadtrueBeta1.271.29MinDomainsInPodTopologySpreadtrueGA1.301.31MinimizeIPTablesRestorefalseAlpha1.261.26MinimizeIPTablesRestoretrueBeta1.271.27MinimizeIPTablesRestoretrueGA1.281.29MixedProtocolLBServicefalseAlpha1.201.23MixedProtocolLBServicetrueBeta1.241.25MixedProtocolLBServicetrueGA1.261.27MountContainersfalseAlpha1.91.16MountContainersfalseDeprecated1.171.17MountPropagationfalseAlpha1.81.9MountPropagationtrueBeta1.101.11MountPropagationtrueGA1.121.14MultiCIDRRangeAllocatorfalseAlpha1.251.28NamespaceDefaultLabelNametrueBeta1.211.21NamespaceDefaultLabelNametrueGA1.221.23NetworkPolicyEndPortfalseAlpha1.211.21NetworkPolicyEndPorttrueBeta1.221.24NetworkPolicyEndPorttrueGA1.251.26NetworkPolicyStatusfalseAlpha1.241.27NewVolumeManagerReconstructionfalseAlpha1.251.26NewVolumeManagerReconstructiontrueBeta1.271.29NewVolumeManagerReconstructiontrueGA1.301.31NodeDisruptionExclusionfalseAlpha1.161.18NodeDisruptionExclusiontrueBeta1.191.20NodeDisruptionExclusiontrueGA1.211.22NodeLeasefalseAlpha1.121.13NodeLeasetrueBeta1.141.16NodeLeasetrueGA1.171.23NodeOutOfServiceVolumeDetachfalseAlpha1.241.25NodeOutOfServiceVolumeDetachtrueBeta1.261.27NodeOutOfServiceVolumeDetachtrueGA1.281.31NonPreemptingPriorityfalseAlpha1.151.18NonPreemptingPrioritytrueBeta1.191.23NonPreemptingPrioritytrueGA1.241.25OpenAPIV3falseAlpha1.231.23OpenAPIV3trueBeta1.241.26OpenAPIV3trueGA1.271.28PDBUnhealthyPodEvictionPolicyfalseAlpha1.261.26PDBUnhealthyPodEvictionPolicytrueBeta1.271.30PDBUnhealthyPodEvictionPolicytrueGA1.311.32PersistentLocalVolumesfalseAlpha1.71.9PersistentLocalVolumestrueBeta1.101.13PersistentLocalVolumestrueGA1.141.16PersistentVolumeLastPhaseTransitionTimefalseAlpha1.281.28PersistentVolumeLastPhaseTransitionTimetrueBeta1.291.30PersistentVolumeLastPhaseTransitionTimetrueGA1.311.32PodAffinityNamespaceSelectorfalseAlpha1.211.21PodAffinityNamespaceSelectortrueBeta1.221.23PodAffinityNamespaceSelectortrueGA1.241.25PodDisruptionBudgetfalseAlpha1.31.4PodDisruptionBudgettrueBeta1.51.20PodDisruptionBudgettrueGA1.211.25PodDisruptionConditionsfalseAlpha1.251.25PodDisruptionConditionstrueBeta1.261.30PodDisruptionConditionstrueGA1.311.33PodHasNetworkConditionfalseAlpha1.251.27PodHostIPsfalseAlpha1.281.28PodHostIPstrueBeta1.291.30PodHostIPstrueGA1.301.31PodOverheadfalseAlpha1.161.17PodOverheadtrueBeta1.181.23PodOverheadtrueGA1.241.25PodPriorityfalseAlpha1.81.10PodPrioritytrueBeta1.111.13PodPrioritytrueGA1.141.18PodReadinessGatesfalseAlpha1.111.11PodReadinessGatestrueBeta1.121.13PodReadinessGatestrueGA1.141.16PodSecurityfalseAlpha1.221.22PodSecuritytrueBeta1.231.24PodSecuritytrueGA1.251.27PodShareProcessNamespacefalseAlpha1.101.11PodShareProcessNamespacetrueBeta1.121.16PodShareProcessNamespacetrueGA1.171.19PreferNominatedNodefalseAlpha1.211.21PreferNominatedNodetrueBeta1.221.23PreferNominatedNodetrueGA1.241.25ProbeTerminationGracePeriodfalseAlpha1.211.21ProbeTerminationGracePeriodfalseBeta1.221.24ProbeTerminationGracePeriodtrueBeta1.251.27ProbeTerminationGracePeriodtrueGA1.281.28ProxyTerminatingEndpointsfalseAlpha1.221.25ProxyTerminatingEndpointstrueBeta1.261.27ProxyTerminatingEndpointstrueGA1.281.29PVCProtectionfalseAlpha1.91.9PVCProtection–Deprecated1.101.10ReadOnlyAPIDataVolumestrueBeta1.81.9ReadOnlyAPIDataVolumes–GA1.101.10ReadWriteOncePodfalseAlpha1.221.26ReadWriteOncePodtrueBeta1.271.28ReadWriteOncePodtrueGA1.291.30RemainingItemCountfalseAlpha1.151.15RemainingItemCounttrueBeta1.161.28RemainingItemCounttrueGA1.291.32RemoveSelfLinkfalseAlpha1.161.19RemoveSelfLinktrueBeta1.201.23RemoveSelfLinktrueGA1.241.29RequestManagementfalseAlpha1.151.16RequestManagement–Deprecated1.171.17ResourceLimitsPriorityFunctionfalseAlpha1.91.18ResourceLimitsPriorityFunction–Deprecated1.191.19ResourceQuotaScopeSelectorsfalseAlpha1.111.11ResourceQuotaScopeSelectorstrueBeta1.121.16ResourceQuotaScopeSelectorstrueGA1.171.18RetroactiveDefaultStorageClassfalseAlpha1.251.25RetroactiveDefaultStorageClasstrueBeta1.261.27RetroactiveDefaultStorageClasstrueGA1.281.28RootCAConfigMapfalseAlpha1.131.19RootCAConfigMaptrueBeta1.201.20RootCAConfigMaptrueGA1.211.22RotateKubeletClientCertificatetrueBeta1.81.18RotateKubeletClientCertificatetrueGA1.191.21RunAsGrouptrueBeta1.141.20RunAsGrouptrueGA1.211.22RuntimeClassfalseAlpha1.121.13RuntimeClasstrueBeta1.141.19RuntimeClasstrueGA1.201.24ScheduleDaemonSetPodsfalseAlpha1.111.11ScheduleDaemonSetPodstrueBeta1.121.16ScheduleDaemonSetPodstrueGA1.171.18SCTPSupportfalseAlpha1.121.18SCTPSupporttrueBeta1.191.19SCTPSupporttrueGA1.201.22SeccompDefaultfalseAlpha1.221.24SeccompDefaulttrueBeta1.251.26SeccompDefaulttrueGA1.271.28SecurityContextDenyfalseAlpha1.271.29SelectorIndexfalseAlpha1.181.18SelectorIndextrueBeta1.191.19SelectorIndextrueGA1.201.25ServerSideApplyfalseAlpha1.141.15ServerSideApplytrueBeta1.161.21ServerSideApplytrueGA1.221.31ServerSideFieldValidationfalseAlpha1.231.24ServerSideFieldValidationtrueBeta1.251.26ServerSideFieldValidationtrueGA1.271.31ServiceAccountIssuerDiscoveryfalseAlpha1.181.19ServiceAccountIssuerDiscoverytrueBeta1.201.20ServiceAccountIssuerDiscoverytrueGA1.211.23ServiceAppProtocolfalseAlpha1.181.18ServiceAppProtocoltrueBeta1.191.19ServiceAppProtocoltrueGA1.201.22ServiceInternalTrafficPolicyfalseAlpha1.211.21ServiceInternalTrafficPolicytrueBeta1.221.25ServiceInternalTrafficPolicytrueGA1.261.27ServiceIPStaticSubrangefalseAlpha1.241.24ServiceIPStaticSubrangetrueBeta1.251.25ServiceIPStaticSubrangetrueGA1.261.27ServiceLBNodePortControlfalseAlpha1.201.21ServiceLBNodePortControltrueBeta1.221.23ServiceLBNodePortControltrueGA1.241.25ServiceLoadBalancerClassfalseAlpha1.211.21ServiceLoadBalancerClasstrueBeta1.221.23ServiceLoadBalancerClasstrueGA1.241.25ServiceLoadBalancerFinalizerfalseAlpha1.151.15ServiceLoadBalancerFinalizertrueBeta1.161.16ServiceLoadBalancerFinalizertrueGA1.171.20ServiceNodeExclusionfalseAlpha1.81.18ServiceNodeExclusiontrueBeta1.191.20ServiceNodeExclusiontrueGA1.211.22ServiceNodePortStaticSubrangefalseAlpha1.271.27ServiceNodePortStaticSubrangetrueBeta1.281.28ServiceNodePortStaticSubrangetrueGA1.291.30ServiceTopologyfalseAlpha1.171.19ServiceTopologyfalseDeprecated1.201.22SetHostnameAsFQDNfalseAlpha1.191.19SetHostnameAsFQDNtrueBeta1.201.21SetHostnameAsFQDNtrueGA1.221.24SkipReadOnlyValidationGCEfalseAlpha1.281.28SkipReadOnlyValidationGCEtrueDeprecated1.291.30StableLoadBalancerNodeSettrueBeta1.271.29StableLoadBalancerNodeSettrueGA1.301.31StartupProbefalseAlpha1.161.17StartupProbetrueBeta1.181.19StartupProbetrueGA1.201.23StatefulSetMinReadySecondsfalseAlpha1.221.22StatefulSetMinReadySecondstrueBeta1.231.24StatefulSetMinReadySecondstrueGA1.251.26StorageObjectInUseProtectiontrueBeta1.101.10StorageObjectInUseProtectiontrueGA1.111.24StreamingProxyRedirectsfalseBeta1.51.5StreamingProxyRedirectstrueBeta1.61.17StreamingProxyRedirectstrueDeprecated1.181.21StreamingProxyRedirectsfalseDeprecated1.221.24SupportIPVSProxyModefalseAlpha1.81.8SupportIPVSProxyModefalseBeta1.91.9SupportIPVSProxyModetrueBeta1.101.10SupportIPVSProxyModetrueGA1.111.20SupportNodePidsLimitfalseAlpha1.141.14SupportNodePidsLimittrueBeta1.151.19SupportNodePidsLimittrueGA1.201.23SupportPodPidsLimitfalseAlpha1.101.13SupportPodPidsLimittrueBeta1.141.19SupportPodPidsLimittrueGA1.201.23SuspendJobfalseAlpha1.211.21SuspendJobtrueBeta1.221.23SuspendJobtrueGA1.241.25SysctlstrueBeta1.111.20SysctlstrueGA1.211.22TaintBasedEvictionsfalseAlpha1.61.12TaintBasedEvictionstrueBeta1.131.17TaintBasedEvictionstrueGA1.181.20TaintNodesByConditionfalseAlpha1.81.11TaintNodesByConditiontrueBeta1.121.16TaintNodesByConditiontrueGA1.171.18TokenRequestfalseAlpha1.101.11TokenRequesttrueBeta1.121.19TokenRequesttrueGA1.201.21TokenRequestProjectionfalseAlpha1.111.11TokenRequestProjectiontrueBeta1.121.19TokenRequestProjectiontrueGA1.201.21TopologyManagerfalseAlpha1.161.17TopologyManagertrueBeta1.181.26TopologyManagertrueGA1.271.28TTLAfterFinishedfalseAlpha1.121.20TTLAfterFinishedtrueBeta1.211.22TTLAfterFinishedtrueGA1.231.24UserNamespacesStatelessPodsSupportfalseAlpha1.251.27ValidateProxyRedirectsfalseAlpha1.121.13ValidateProxyRedirectstrueBeta1.141.21ValidateProxyRedirectstrueDeprecated1.221.24ValidatingAdmissionPolicyfalseAlpha1.261.27ValidatingAdmissionPolicyfalseBeta1.281.29ValidatingAdmissionPolicytrueGA1.301.31VolumeCapacityPriorityfalseAlpha1.211.32VolumePVCDataSourcefalseAlpha1.151.15VolumePVCDataSourcetrueBeta1.161.17VolumePVCDataSourcetrueGA1.181.21VolumeSchedulingfalseAlpha1.91.9VolumeSchedulingtrueBeta1.101.12VolumeSchedulingtrueGA1.131.16VolumeSnapshotDataSourcefalseAlpha1.121.16VolumeSnapshotDataSourcetrueBeta1.171.19VolumeSnapshotDataSourcetrueGA1.201.22VolumeSubpathtrueGA1.101.24VolumeSubpathEnvExpansionfalseAlpha1.141.14VolumeSubpathEnvExpansiontrueBeta1.151.16VolumeSubpathEnvExpansiontrueGA1.171.24WarningHeaderstrueBeta1.191.21WarningHeaderstrueGA1.221.24WatchBookmarkfalseAlpha1.151.15WatchBookmarktrueBeta1.161.16WatchBookmarktrueGA1.171.32WindowsEndpointSliceProxyingfalseAlpha1.191.20WindowsEndpointSliceProxyingtrueBeta1.211.21WindowsEndpointSliceProxyingtrueGA1.221.24WindowsGMSAfalseAlpha1.141.15WindowsGMSAtrueBeta1.161.17WindowsGMSAtrueGA1.181.18WindowsHostProcessContainersfalseAlpha1.221.22WindowsHostProcessContainerstrueBeta1.231.25WindowsHostProcessContainerstrueGA1.261.27WindowsRunAsUserNamefalseAlpha1.161.16WindowsRunAsUserNametrueBeta1.171.17WindowsRunAsUserNametrueGA1.181.20ZeroLimitedNominalConcurrencySharesfalseBeta1.291.29ZeroLimitedNominalConcurrencySharestrueGA1.301.31Descriptions for removed feature gatesAccelerators: Provided an early form of plugin to enable Nvidia GPU support when using Docker Engine; no longer available. See Device Plugins for an alternative.AdmissionWebhookMatchConditions: Enable match conditions on mutating & validating admission webhooks.AdvancedAuditing: Enable advanced auditingAffinityInAnnotations: Enable setting Pod affinity or anti-affinity.AggregatedDiscoveryEndpoint: Enable a single HTTP endpoint /discovery/<version> which supports native HTTP caching with ETags containing all APIResources known to the API server.AllowExtTrafficLocalEndpoints: Enable a service to route external requests to node local endpoints.AllowInsecureBackendProxy: Enable the users to skip TLS verification of kubelets on Pod log requests.APIListChunking: Enable the API clients to retrieve (LIST or GET) resources from API server in chunks.APIPriorityAndFairness: Enable managing request concurrency with prioritization and fairness at each server. (Renamed from RequestManagement)APISelfSubjectReview: Activate the SelfSubjectReview API which allows users to see the requesting subject's authentication information. See API access to authentication information for a client for more details.AppArmor: Enable use of AppArmor mandatory access control for Pods running on Linux nodes. See AppArmor Tutorial for more details.AppArmorFields:Enable AppArmor related security context settings.For more information about AppArmor and Kubernetes, read the AppArmor section within security features in the Linux kernel.AttachVolumeLimit: Enable volume plugins to report limits on number of volumes that can be attached to a node. See dynamic volume limits for more details.BalanceAttachedNodeVolumes: Include volume count on node to be considered for balanced resource allocation while scheduling. A node which has closer CPU, memory utilization, and volume count is favored by the scheduler while making decisions.BlockVolume: Enable the definition and consumption of raw block devices in Pods. See Raw Block Volume Support for more details.BoundServiceAccountTokenVolume:Migrate ServiceAccount volumes to use a projected volume consisting of a ServiceAccountTokenVolumeProjection. Cluster admins can use metric serviceaccount_stale_tokens_total to monitor workloads that are depending on the extended tokens. If there are no such workloads, turn off extended tokens by starting kube-apiserver with flag --service-account-extend-token-expiration=false.Check Bound Service Account Tokens for more details.CloudDualStackNodeIPs: Enables dual-stack kubelet --node-ip with external cloud providers. See Configure IPv4/IPv6 dual-stack for more details.ConfigurableFSGroupPolicy: Allows user to configure volume permission change policy for fsGroups when mounting a volume in a Pod. See Configure volume permission and ownership change policy for Pods for more details.ConsistentHTTPGetHandlers: Normalize HTTP get URL and Header passing for lifecycle handlers with probers.ControllerManagerLeaderMigration: Enables Leader Migration for kube-controller-manager and cloud-controller-manager which allows a cluster operator to live migrate controllers from the kube-controller-manager into an external controller-manager (e.g. the cloud-controller-manager) in an HA cluster without downtime.CPUManager: Enable container level CPU affinity support, see CPU Management Policies.CRIContainerLogRotation: Enable container log rotation for CRI container runtime. The default max size of a log file is 10MB and the default max number of log files allowed for a container is 5. These values can be configured in the kubelet config. See logging at node level for more details.CronJobControllerV2: Use an alternative implementation of the CronJob controller. Otherwise, version 1 of the same controller is selected.CronJobTimeZone: Allow the use of the timeZone optional field in CronJobsCSIBlockVolume: Enable external CSI volume drivers to support block storage. See csi raw block volume support for more details.CSIDriverRegistry: Enable all logic related to the CSIDriver API object in csi.storage.k8s.io.CSIInlineVolume: Enable CSI Inline volumes support for pods.CSIMigration: Enables shims and translation logic to route volume operations from in-tree plugins to corresponding pre-installed CSI pluginsCSIMigrationAWS: Enables shims and translation logic to route volume operations from the AWS-EBS in-tree plugin to EBS CSI plugin. Supports falling back to in-tree EBS plugin for mount operations to nodes that have the feature disabled or that do not have EBS CSI plugin installed and configured. Does not support falling back for provision operations, for those the CSI plugin must be installed and configured.CSIMigrationAWSComplete: Stops registering the EBS in-tree plugin in kubelet and volume controllers and enables shims and translation logic to route volume operations from the AWS-EBS in-tree plugin to EBS CSI plugin. Requires CSIMigration and CSIMigrationAWS feature flags enabled and EBS CSI plugin installed and configured on all nodes in the cluster. This flag has been deprecated in favor of the InTreePluginAWSUnregister feature flag which prevents the registration of in-tree EBS plugin.CSIMigrationAzureDisk: Enables shims and translation logic to route volume operations from the Azure-Disk in-tree plugin to AzureDisk CSI plugin. Supports falling back to in-tree AzureDisk plugin for mount operations to nodes that have the feature disabled or that do not have AzureDisk CSI plugin installed and configured. Does not support falling back for provision operations, for those the CSI plugin must be installed and configured. Requires CSIMigration feature flag enabled.CSIMigrationAzureDiskComplete: Stops registering the Azure-Disk in-tree plugin in kubelet and volume controllers and enables shims and translation logic to route volume operations from the Azure-Disk in-tree plugin to AzureDisk CSI plugin. Requires CSIMigration and CSIMigrationAzureDisk feature flags enabled and AzureDisk CSI plugin installed and configured on all nodes in the cluster. This flag has been deprecated in favor of the InTreePluginAzureDiskUnregister feature flag which prevents the registration of in-tree AzureDisk plugin.CSIMigrationAzureFile: Enables shims and translation logic to route volume operations from the Azure-File in-tree plugin to AzureFile CSI plugin. Supports falling back to in-tree AzureFile plugin for mount operations to nodes that have the feature disabled or that do not have AzureFile CSI plugin installed and configured. Does not support falling back for provision operations, for those the CSI plugin must be installed and configured. Requires CSIMigration feature flag enabled.CSIMigrationAzureFileComplete: Stops registering the Azure-File in-tree plugin in kubelet and volume controllers and enables shims and translation logic to route volume operations from the Azure-File in-tree plugin to AzureFile CSI plugin. Requires CSIMigration and CSIMigrationAzureFile feature flags enabled and AzureFile CSI plugin installed and configured on all nodes in the cluster. This flag has been deprecated in favor of the InTreePluginAzureFileUnregister feature flag which prevents the registration of in-tree AzureFile plugin.CSIMigrationGCE: Enables shims and translation logic to route volume operations from the GCE-PD in-tree plugin to PD CSI plugin. Supports falling back to in-tree GCE plugin for mount operations to nodes that have the feature disabled or that do not have PD CSI plugin installed and configured. Does not support falling back for provision operations, for those the CSI plugin must be installed and configured. Requires CSIMigration feature flag enabled.CSIMigrationGCEComplete: Stops registering the GCE-PD in-tree plugin in kubelet and volume controllers and enables shims and translation logic to route volume operations from the GCE-PD in-tree plugin to PD CSI plugin. Requires CSIMigration and CSIMigrationGCE feature flags enabled and PD CSI plugin installed and configured on all nodes in the cluster. This flag has been deprecated in favor of the InTreePluginGCEUnregister feature flag which prevents the registration of in-tree GCE PD plugin.CSIMigrationOpenStack: Enables shims and translation logic to route volume operations from the Cinder in-tree plugin to Cinder CSI plugin. Supports falling back to in-tree Cinder plugin for mount operations to nodes that have the feature disabled or that do not have Cinder CSI plugin installed and configured. Does not support falling back for provision operations, for those the CSI plugin must be installed and configured. Requires CSIMigration feature flag enabled.CSIMigrationOpenStackComplete: Stops registering the Cinder in-tree plugin in kubelet and volume controllers and enables shims and translation logic to route volume operations from the Cinder in-tree plugin to Cinder CSI plugin. Requires CSIMigration and CSIMigrationOpenStack feature flags enabled and Cinder CSI plugin installed and configured on all nodes in the cluster. This flag has been deprecated in favor of the InTreePluginOpenStackUnregister feature flag which prevents the registration of in-tree openstack cinder plugin.CSIMigrationRBD:Enables shims and translation logic to route volume operations from the RBD in-tree plugin to Ceph RBD CSI plugin. Requires CSIMigration and csiMigrationRBD feature flags enabled and Ceph CSI plugin installed and configured in the cluster.This feature gate was deprecated in favor of the InTreePluginRBDUnregister feature gate, which prevents the registration of in-tree RBD plugin.CSIMigrationvSphere: Enables shims and translation logic to route volume operations from the vSphere in-tree plugin to vSphere CSI plugin. Supports falling back to in-tree vSphere plugin for mount operations to nodes that have the feature disabled or that do not have vSphere CSI plugin installed and configured. Does not support falling back for provision operations, for those the CSI plugin must be installed and configured. Requires CSIMigration feature flag enabled.CSIMigrationvSphereComplete: Stops registering the vSphere in-tree plugin in kubelet and volume controllers and enables shims and translation logic to route volume operations from the vSphere in-tree plugin to vSphere CSI plugin. Requires CSIMigration and CSIMigrationvSphere feature flags enabled and vSphere CSI plugin installed and configured on all nodes in the cluster. This flag has been deprecated in favor of the InTreePluginvSphereUnregister feature flag which prevents the registration of in-tree vsphere plugin.CSINodeExpandSecret: Enable passing secret authentication data to a CSI driver for use during a NodeExpandVolume CSI operation.CSINodeInfo: Enable all logic related to the CSINodeInfo API object in csi.storage.k8s.io.CSIPersistentVolume: Enable discovering and mounting volumes provisioned through a CSI (Container Storage Interface) compatible volume plugin.CSIServiceAccountToken: Enable CSI drivers to receive the pods' service account token that they mount volumes for. See Token Requests.CSIStorageCapacity: Enables CSI drivers to publish storage capacity information and the Kubernetes scheduler to use that information when scheduling pods. See Storage Capacity. Check the csi volume type documentation for more details.CSIVolumeFSGroupPolicy: Allows CSIDrivers to use the fsGroupPolicy field. This field controls whether volumes created by a CSIDriver support volume ownership and permission modifications when these volumes are mounted.CSRDuration: Allows clients to request a duration for certificates issued via the Kubernetes CSR API.CustomPodDNS: Enable customizing the DNS settings for a Pod using its dnsConfig property. Check Pod's DNS Config for more details.CustomResourceDefaulting: Enable CRD support for default values in OpenAPI v3 validation schemas.CustomResourcePublishOpenAPI: Enables publishing of CRD OpenAPI specs.CustomResourceSubresources: Enable /status and /scale subresources on resources created from CustomResourceDefinition.CustomResourceValidation: Enable schema based validation on resources created from CustomResourceDefinition.CustomResourceValidationExpressions: Enable expression language validation in CRD which will validate customer resource based on validation rules written in the x-kubernetes-validations extension.CustomResourceWebhookConversion: Enable webhook-based conversion on resources created from CustomResourceDefinition.DaemonSetUpdateSurge: Enables the DaemonSet workloads to maintain availability during update per node. See Perform a Rolling Update on a DaemonSet.DefaultHostNetworkHostPortsInPodTemplates:This feature gate controls the point at which a default value for .spec.containers[*].ports[*].hostPort is assigned, for Pods using hostNetwork: true. The default since Kubernetes v1.28 is to only set a default value in Pods.Enabling this means a default will be assigned even to the .spec of an embedded PodTemplate (for example, in a Deployment), which is the way that older releases of Kubernetes worked. You should migrate your code so that it does not rely on the legacy behavior.DefaultPodTopologySpread: Enables the use of PodTopologySpread scheduling plugin to do default spreading.DelegateFSGroupToCSIDriver: If supported by the CSI driver, delegates the role of applying fsGroup from a Pod's securityContext to the driver by passing fsGroup through the NodeStageVolume and NodePublishVolume CSI calls.DevicePluginCDIDevices: Enable support to CDI device IDs in the Device Plugin API.DevicePlugins: Enable the device-plugins based resource provisioning on nodes.DisableAcceleratorUsageMetrics: Disable accelerator metrics collected by the kubelet.DisableCloudProviders:Enabling this feature gate deactivated functionality in kube-apiserver, kube-controller-manager and kubelet that related to the --cloud-provider command line argument.In Kubernetes v1.31 and later, the only valid values for --cloud-provider are the empty string (no cloud provider integration), or "external" (integration via a separate cloud-controller-manager).DisableKubeletCloudCredentialProviders: Enabling the feature gate deactivated the legacy in-tree functionality within the kubelet, that allowed the kubelet to to authenticate to a cloud provider container registry for container image pulls.DownwardAPIHugePages: Enables usage of hugepages in downward API.DRAControlPlaneController: Enables support for resources with custom parameters and a lifecycle that is independent of a Pod. Allocation of resources is handled by a resource driver's control plane controller.DryRun: Enable server-side dry run requests so that validation, merging, and mutation can be tested without committing.DynamicAuditing: Used to enable dynamic auditing before v1.19.DynamicKubeletConfig: Enable the dynamic configuration of kubelet. The feature is no longer supported outside of supported skew policy. The feature gate was removed from kubelet in 1.24.DynamicProvisioningScheduling: Extend the default scheduler to be aware of volume topology and handle PV provisioning. This feature was superseded by the VolumeScheduling feature in v1.12.DynamicVolumeProvisioning: Enable the dynamic provisioning of persistent volumes to Pods.EfficientWatchResumption: Allows for storage-originated bookmark (progress notify) events to be delivered to the users. This is only applied to watch operations.EnableAggregatedDiscoveryTimeout: Enable the five second timeout on aggregated discovery calls.EnableEquivalenceClassCache: Enable the scheduler to cache equivalence of nodes when scheduling Pods.EndpointSlice: Enables EndpointSlices for more scalable and extensible network endpoints. See Enabling EndpointSlices.EndpointSliceNodeName: Enables EndpointSlice nodeName field.EndpointSliceProxying: When enabled, kube-proxy running on Linux will use EndpointSlices as the primary data source instead of Endpoints, enabling scalability and performance improvements. See Enabling Endpoint Slices.EndpointSliceTerminatingCondition: Enables EndpointSlice terminating and serving condition fields.EphemeralContainers: Enable the ability to add ephemeral containers to running Pods.EvenPodsSpread: Enable pods to be scheduled evenly across topology domains. See Pod Topology Spread Constraints.ExpandCSIVolumes: Enable the expanding of CSI volumes.ExpandedDNSConfig: Enable kubelet and kube-apiserver to allow more DNS search paths and longer list of DNS search paths. This feature requires container runtime support(Containerd: v1.5.6 or higher, CRI-O: v1.22 or higher). See Expanded DNS Configuration.ExpandInUsePersistentVolumes: Enable expanding in-use PVCs. See Resizing an in-use PersistentVolumeClaim.ExpandPersistentVolumes: Enable the expanding of persistent volumes. See Expanding Persistent Volumes Claims.ExperimentalCriticalPodAnnotation: Enable annotating specific pods as critical so that their scheduling is guaranteed. This feature is deprecated by Pod Priority and Preemption as of v1.13.ExperimentalHostUserNamespaceDefaulting: Enabling the defaulting user namespace to host. This is for containers that are using other host namespaces, host mounts, or containers that are privileged or using specific non-namespaced capabilities (e.g. MKNODE, SYS_MODULE etc.). This should only be enabled if user namespace remapping is enabled in the Docker daemon.ExternalPolicyForExternalIP: Fix a bug where ExternalTrafficPolicy is not applied to Service ExternalIPs.GCERegionalPersistentDisk: Enable the regional PD feature on GCE.GenericEphemeralVolume: Enables ephemeral, inline volumes that support all features of normal volumes (can be provided by third-party storage vendors, storage capacity tracking, restore from snapshot, etc.). See Ephemeral Volumes.GRPCContainerProbe: Enables the gRPC probe method for {Liveness,Readiness,Startup}Probe. See Configure Liveness, Readiness and Startup Probes.HPAContainerMetrics: Allow HorizontalPodAutoscalers to scale based on metrics from individual containers within target pods.HugePages: Enable the allocation and consumption of pre-allocated huge pages.HugePageStorageMediumSize: Enable support for multiple sizes pre-allocated huge pages.HyperVContainer: Enable Hyper-V isolation for Windows containers.IdentifyPodOS: Allows the Pod OS field to be specified. This helps in identifying the OS of the pod authoritatively during the API server admission time.ImmutableEphemeralVolumes: Allows for marking individual Secrets and ConfigMaps as immutable for better safety and performance.IndexedJob: Allows the Job controller to manage Pod completions per completion index.IngressClassNamespacedParams: Allow namespace-scoped parameters reference in IngressClass resource. This feature adds two fields - Scope and Namespace to IngressClass.spec.parameters.Initializers: Allow asynchronous coordination of object creation using the Initializers admission plugin.InTreePluginAWSUnregister: Stops registering the aws-ebs in-tree plugin in kubelet and volume controllers.InTreePluginAzureDiskUnregister: Stops registering the azuredisk in-tree plugin in kubelet and volume controllers.InTreePluginAzureFileUnregister: Stops registering the azurefile in-tree plugin in kubelet and volume controllers.InTreePluginGCEUnregister: Stops registering the gce-pd in-tree plugin in kubelet and volume controllers.InTreePluginOpenStackUnregister: Stops registering the OpenStack cinder in-tree plugin in kubelet and volume controllers.InTreePluginRBDUnregister: Stops registering the RBD in-tree plugin within kubelet and volume controllers.InTreePluginvSphereUnregister: Stops registering the vSphere in-tree plugin in kubelet and volume controllers.IPTablesOwnershipCleanup: This causes kubelet to no longer create legacy iptables rules.IPv6DualStack: Enable dual stack support for IPv6.JobMutableNodeSchedulingDirectives: Allows updating node scheduling directives in the pod template of Job.JobPodFailurePolicy: Allow users to specify handling of pod failures based on container exit codes and pod conditions.JobReadyPods: Enables tracking the number of Pods that have a Ready condition. The count of Ready pods is recorded in the status of a Job status.JobTrackingWithFinalizers: Enables tracking Job completions without relying on Pods remaining in the cluster indefinitely. The Job controller uses Pod finalizers and a field in the Job status to keep track of the finished Pods to count towards completion.KMSv2: Enables KMS v2 API for encryption at rest. See Using a KMS Provider for data encryption for more details.KMSv2KDF: Enables KMS v2 to generate single use data encryption keys. See Using a KMS Provider for data encryption for more details. If the KMSv2 feature gate is not enabled in your cluster, the value of the KMSv2KDF feature gate has no effect.KubeletConfigFile: Enable loading kubelet configuration from a file specified using a config file. See setting kubelet parameters via a config file for more details.KubeletCredentialProviders: Enable kubelet exec credential providers for image pull credentials.KubeletPluginsWatcher: Enable probe-based plugin watcher utility to enable kubelet to discover plugins such as CSI volume drivers.KubeletPodResources: Enable the kubelet's pod resources gRPC endpoint. See Support Device Monitoring for more details.KubeletPodResourcesGetAllocatable: Enable the kubelet's pod resources GetAllocatableResources functionality. This API augments the resource allocation reportingKubeProxyDrainingTerminatingNodes: Implement connection draining for terminating nodes for externalTrafficPolicy: Cluster services.LegacyNodeRoleBehavior: When disabled, legacy behavior in service load balancers and node disruption will ignore the node-role.kubernetes.io/master label in favor of the feature-specific labels provided by NodeDisruptionExclusion and ServiceNodeExclusion.LegacyServiceAccountTokenCleanUp: Enable cleaning up Secret-based service account tokens when they are not used in a specified time (default to be one year).LegacyServiceAccountTokenNoAutoGeneration: Stop auto-generation of Secret-based service account tokens.LegacyServiceAccountTokenTracking: Track usage of Secret-based service account tokens.LocalStorageCapacityIsolation: Enable the consumption of local ephemeral storage and also the sizeLimit property of an emptyDir volume.MinDomainsInPodTopologySpread: Enable minDomains in Pod topology spread constraints.MinimizeIPTablesRestore: Enables new performance improvement logics in the kube-proxy iptables mode.MixedProtocolLBService: Enable using different protocols in the same LoadBalancer type Service instance.MountContainers: Enable using utility containers on host as the volume mounter.MountPropagation: Enable sharing volume mounted by one container to other containers or pods. For more details, please see mount propagation.MultiCIDRRangeAllocator: Enables the MultiCIDR range allocator.NamespaceDefaultLabelName: Configure the API Server to set an immutable label kubernetes.io/metadata.name on all namespaces, containing the namespace name.NetworkPolicyEndPort: Allows you to define ports in a NetworkPolicy rule as a range of port numbers.NetworkPolicyStatus: Enable the status subresource for NetworkPolicy objects.NewVolumeManagerReconstruction:Enables improved discovery of mounted volumes during kubelet startup. Since the associated code had been significantly refactored, Kubernetes versions 1.25 to 1.29 allowed you to opt-out in case the kubelet got stuck at the startup, or did not unmount volumes from terminated Pods.This refactoring was behind the SELinuxMountReadWriteOncePod feature gate in Kubernetes releases 1.25 and 1.26.NodeDisruptionExclusion: Enable use of the Node label node.kubernetes.io/exclude-disruption which prevents nodes from being evacuated during zone failures.NodeLease: Enable the new Lease API to report node heartbeats, which could be used as a node health signal.NodeOutOfServiceVolumeDetach: When a Node is marked out-of-service using the node.kubernetes.io/out-of-service taint, Pods on the node will be forcefully deleted if they can not tolerate this taint, and the volume detach operations for Pods terminating on the node will happen immediately. The deleted Pods can recover quickly on different nodes.NonPreemptingPriority: Enable preemptionPolicy field for PriorityClass and Pod.OpenAPIV3: Enables the API server to publish OpenAPI v3.PDBUnhealthyPodEvictionPolicy: Enables the unhealthyPodEvictionPolicy field of a PodDisruptionBudget. This specifies when unhealthy pods should be considered for eviction. Please see Unhealthy Pod Eviction Policy for more details.PersistentLocalVolumes: Enable the usage of local volume type in Pods. Pod affinity has to be specified if requesting a local volume.PersistentVolumeLastPhaseTransitionTime: Adds a new field to PersistentVolume which holds a timestamp of when the volume last transitioned its phase.PodAffinityNamespaceSelector: Enable the Pod Affinity Namespace Selector and CrossNamespacePodAffinity quota scope features.PodDisruptionBudget: Enable the PodDisruptionBudget feature.PodDisruptionConditions: Enabled support for appending a dedicated pod condition indicating that the pod is being deleted due to a disruption.PodHasNetworkCondition: Enable the kubelet to mark the PodHasNetwork condition on pods. This was renamed to PodReadyToStartContainersCondition in 1.28.PodHostIPs: Enable the status.hostIPs field for pods and the downward API. The field lets you expose host IP addresses to workloads.PodOverhead: Enable the PodOverhead feature to account for pod overheads.PodPriority: Enable the descheduling and preemption of Pods based on their priorities.PodReadinessGates: Enable the setting of PodReadinessGate field for extending Pod readiness evaluation. See Pod readiness gate for more details.PodSecurity: Enables the PodSecurity admission plugin.PodShareProcessNamespace: Enable the setting of shareProcessNamespace in a Pod for sharing a single process namespace between containers running in a pod. More details can be found in Share Process Namespace between Containers in a Pod.PreferNominatedNode: This flag tells the scheduler whether the nominated nodes will be checked first before looping through all the other nodes in the cluster.ProbeTerminationGracePeriod: Enable setting probe-level terminationGracePeriodSeconds on pods. See the enhancement proposal for more details.ProxyTerminatingEndpoints: Enable the kube-proxy to handle terminating endpoints when ExternalTrafficPolicy=Local.PVCProtection: Enable the prevention of a PersistentVolumeClaim (PVC) from being deleted when it is still used by any Pod.ReadOnlyAPIDataVolumes:Set configMap, secret, downwardAPI and projected volumes to be mounted read-only.Since Kubernetes v1.10, these volume types are always read-only and you cannot opt out.ReadWriteOncePod: Enables the usage of ReadWriteOncePod PersistentVolume access mode.RemainingItemCount: Allow the API servers to show a count of remaining items in the response to a chunking list request.RemoveSelfLink: Sets the .metadata.selfLink field to blank (empty string) for all objects and collections. This field has been deprecated since the Kubernetes v1.16 release. When this feature is enabled, the .metadata.selfLink field remains part of the Kubernetes API, but is always unset.RequestManagement: Enables managing request concurrency with prioritization and fairness at each API server. Deprecated by APIPriorityAndFairness since 1.17.ResourceLimitsPriorityFunction: Enable a scheduler priority function that assigns a lowest possible score of 1 to a node that satisfies at least one of the input Pod's cpu and memory limits. The intent is to break ties between nodes with same scores.ResourceQuotaScopeSelectors: Enable resource quota scope selectors.RetroactiveDefaultStorageClass: Allow assigning StorageClass to unbound PVCs retroactively.RootCAConfigMap: Configure the kube-controller-manager to publish a ConfigMap named kube-root-ca.crt to every namespace. This ConfigMap contains a CA bundle used for verifying connections to the kube-apiserver. See Bound Service Account Tokens for more details.RotateKubeletClientCertificate: Enable the rotation of the client TLS certificate on the kubelet. See kubelet configuration for more details.RunAsGroup: Enable control over the primary group ID set on the init processes of containers.RuntimeClass: Enable the RuntimeClass feature for selecting container runtime configurations.ScheduleDaemonSetPods: Enable DaemonSet Pods to be scheduled by the default scheduler instead of the DaemonSet controller.SCTPSupport: Enables the SCTP protocol value in Pod, Service, Endpoints, EndpointSlice, and NetworkPolicy definitions.SeccompDefault: Enables the use of RuntimeDefault as the default seccomp profile for all workloads. The seccomp profile is specified in the securityContext of a Pod and/or a Container.SecurityContextDeny: This gate signals that the SecurityContextDeny admission controller is deprecated.SelectorIndex: Allows label and field based indexes in API server watch cache to accelerate list operations.ServerSideApply: Enables the Sever Side Apply (SSA) feature on the API Server.ServerSideFieldValidation: Enables server-side field validation. This means the validation of resource schema is performed at the API server side rather than the client side (for example, the kubectl create or kubectl apply command line).ServiceAccountIssuerDiscovery: Enable OIDC discovery endpoints (issuer and JWKS URLs) for the service account issuer in the API server. See Configure Service Accounts for Pods for more details.ServiceAppProtocol: Enables the appProtocol field on Services and Endpoints.ServiceInternalTrafficPolicy: Enables the internalTrafficPolicy field on ServicesServiceIPStaticSubrange: Enables a strategy for Services ClusterIP allocations, whereby the ClusterIP range is subdivided. Dynamic allocated ClusterIP addresses will be allocated preferently from the upper range allowing users to assign static ClusterIPs from the lower range with a low risk of collision. See Avoiding collisions for more details.ServiceLBNodePortControl: Enables the allocateLoadBalancerNodePorts field on Services.ServiceLoadBalancerClass: Enables the loadBalancerClass field on Services. See Specifying class of load balancer implementation for more details.ServiceLoadBalancerFinalizer: Enable finalizer protection for Service load balancers.ServiceNodeExclusion: Enable the exclusion of nodes from load balancers created by a cloud provider. A node is eligible for exclusion if labelled with "node.kubernetes.io/exclude-from-external-load-balancers".ServiceNodePortStaticSubrange: Enables the use of different port allocation strategies for NodePort Services. For more details, see reserve NodePort ranges to avoid collisions.ServiceTopology: Enable service to route traffic based upon the Node topology of the cluster.SetHostnameAsFQDN: Enable the ability of setting Fully Qualified Domain Name(FQDN) as the hostname of a pod. See Pod's setHostnameAsFQDN field.SkipReadOnlyValidationGCE: Skip validation that GCE PersistentDisk volumes are in read-only mode.StableLoadBalancerNodeSet: Enables less load balancer re-configurations by the service controller (KCCM) as an effect of changing node state.StartupProbe: Enable the startup probe in the kubelet.StatefulSetMinReadySeconds: Allows minReadySeconds to be respected by the StatefulSet controller.StorageObjectInUseProtection: Postpone the deletion of PersistentVolume or PersistentVolumeClaim objects if they are still being used.StreamingProxyRedirects: Instructs the API server to intercept (and follow) redirects from the backend (kubelet) for streaming requests. Examples of streaming requests include the exec, attach and port-forward requests.SupportIPVSProxyMode: Enable providing in-cluster service load balancing using IPVS. See service proxies for more details.SupportNodePidsLimit: Enable the support to limiting PIDs on the Node. The parameter pid=<number> in the --system-reserved and --kube-reserved options can be specified to ensure that the specified number of process IDs will be reserved for the system as a whole and for Kubernetes system daemons respectively.SupportPodPidsLimit: Enable the support to limiting PIDs in Pods.SuspendJob: Enable support to suspend and resume Jobs. For more details, see the Jobs docs.Sysctls: Enable support for namespaced kernel parameters (sysctls) that can be set for each pod. See sysctls for more details.TaintBasedEvictions: Enable evicting pods from nodes based on taints on Nodes and tolerations on Pods. See taints and tolerations for more details.TaintNodesByCondition: Enable automatic tainting nodes based on node conditions.TokenRequest: Enable the TokenRequest endpoint on service account resources.TokenRequestProjection: Enable the injection of service account tokens into a Pod through a projected volume.TopologyManager: Enable a mechanism to coordinate fine-grained hardware resource assignments for different components in Kubernetes. See Control Topology Management Policies on a node.TTLAfterFinished: Allow a TTL controller to clean up resources after they finish execution.UserNamespacesStatelessPodsSupport: Enable user namespace support for stateless Pods. This feature gate was superseded by the UserNamespacesSupport feature gate in the Kubernetes v1.28 release.ValidateProxyRedirects: This flag controls whether the API server should validate that redirects are only followed to the same host. Only used if the StreamingProxyRedirects flag is enabled.ValidatingAdmissionPolicy: Enable ValidatingAdmissionPolicy support for CEL validations be used in Admission Control.VolumeCapacityPriority: Enable support for prioritizing nodes in different topologies based on available PV capacity. This feature is renamed to StorageCapacityScoring in v1.33.VolumePVCDataSource: Enable support for specifying an existing PVC as a DataSource.VolumeScheduling: Enable volume topology aware scheduling and make the PersistentVolumeClaim (PVC) binding aware of scheduling decisions. It also enables the usage of local volume type when used together with the PersistentLocalVolumes feature gate.VolumeSnapshotDataSource: Enable volume snapshot data source support.VolumeSubpath: Allow mounting a subpath of a volume in a container.VolumeSubpathEnvExpansion: Enable subPathExpr field for expanding environment variables into a subPath.WarningHeaders: Allow sending warning headers in API responses.WatchBookmark: Enable support for watch bookmark events.WindowsEndpointSliceProxying: When enabled, kube-proxy running on Windows will use EndpointSlices as the primary data source instead of Endpoints, enabling scalability and performance improvements. See Enabling Endpoint Slices.WindowsGMSA: Enables passing of GMSA credential specs from pods to container runtimes.WindowsHostProcessContainers: Enables support for Windows HostProcess containers.WindowsRunAsUserName: Enable support for running applications in Windows containers with as a non-default user. See Configuring RunAsUserName for more details.ZeroLimitedNominalConcurrencyShares: Allow priority & fairness in the API server to use a zero value for the nominalConcurrencyShares field of the limited section of a priority level.


# kubectl completion

kubectl completionSynopsisOutput shell completion code for the specified shell (bash, zsh, fish, or powershell). The shell code must be evaluated to provide interactive completion of kubectl commands. This can be done by sourcing it from the .bash_profile.Detailed instructions on how to do this are available here: for macOS: https://kubernetes.io/docs/tasks/tools/install-kubectl-macos/#enable-shell-autocompletion for linux: https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/#enable-shell-autocompletion for windows: https://kubernetes.io/docs/tasks/tools/install-kubectl-windows/#enable-shell-autocompletion Note for zsh users: [1] zsh completions are only supported in versions of zsh >= 5.2.kubectl completion SHELL Examples # Installing bash completion on macOS using homebrew ## If running Bash 3.2 included with macOS brew install bash-completion ## or, if running Bash 4.1+ brew install bash-completion@2 ## If kubectl is installed via homebrew, this should start working immediately ## If you've installed via other means, you may need add the completion to your completion directory kubectl completion bash > $(brew --prefix)/etc/bash_completion.d/kubectl # Installing bash completion on Linux ## If bash-completion is not installed on Linux, install the 'bash-completion' package ## via your distribution's package manager. ## Load the kubectl completion code for bash into the current shell source <(kubectl completion bash) ## Write bash completion code to a file and source it from .bash_profile kubectl completion bash > ~/.kube/completion.bash.inc printf " # kubectl shell completion source '$HOME/.kube/completion.bash.inc' " >> $HOME/.bash_profile source $HOME/.bash_profile # Load the kubectl completion code for zsh[1] into the current shell source <(kubectl completion zsh) # Set the kubectl completion code for zsh[1] to autoload on startup kubectl completion zsh > "${fpath[1]}/_kubectl" # Load the kubectl completion code for fish[2] into the current shell kubectl completion fish | source # To load completions for each session, execute once: kubectl completion fish > ~/.config/fish/completions/kubectl.fish # Load the kubectl completion code for powershell into the current shell kubectl completion powershell | Out-String | Invoke-Expression # Set kubectl completion code for powershell to run on startup ## Save completion code to a script and execute in the profile kubectl completion powershell > $HOME\.kube\completion.ps1 Add-Content $PROFILE "$HOME\.kube\completion.ps1" ## Execute completion code in the profile Add-Content $PROFILE "if (Get-Command kubectl -ErrorAction SilentlyContinue) { kubectl completion powershell | Out-String | Invoke-Expression }" ## Add completion code directly to the $PROFILE script kubectl completion powershell >> $PROFILE Options-h, --helphelp for completionParent Options Inherited--as stringUsername to impersonate for the operation. User could be a regular user or a service account in a namespace.--as-group stringsGroup to impersonate for the operation, this flag can be repeated to specify multiple groups.--as-uid stringUID to impersonate for the operation.--cache-dir string Default: "$HOME/.kube/cache"Default cache directory--certificate-authority stringPath to a cert file for the certificate authority--client-certificate stringPath to a client certificate file for TLS--client-key stringPath to a client key file for TLS--cluster stringThe name of the kubeconfig cluster to use--context stringThe name of the kubeconfig context to use--disable-compressionIf true, opt-out of response compression for all requests to the server--insecure-skip-tls-verifyIf true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure--kubeconfig stringPath to the kubeconfig file to use for CLI requests.--kuberc stringPath to the kuberc file to use for preferences. This can be disabled by exporting KUBECTL_KUBERC=false feature gate or turning off the feature KUBERC=off.--match-server-versionRequire server version to match client version-n, --namespace stringIf present, the namespace scope for this CLI request--password stringPassword for basic authentication to the API server--profile string Default: "none"Name of profile to capture. One of (none|cpu|heap|goroutine|threadcreate|block|mutex)--profile-output string Default: "profile.pprof"Name of the file to write the profile to--request-timeout string Default: "0"The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests.-s, --server stringThe address and port of the Kubernetes API server--storage-driver-buffer-duration duration Default: 1m0sWrites in the storage driver will be buffered for this duration, and committed to the non memory backends as a single transaction--storage-driver-db string Default: "cadvisor"database name--storage-driver-host string Default: "localhost:8086"database host:port--storage-driver-password string Default: "root"database password--storage-driver-secureuse secure connection with database--storage-driver-table string Default: "stats"table name--storage-driver-user string Default: "root"database username--tls-server-name stringServer name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used--token stringBearer token for authentication to the API server--user stringThe name of the kubeconfig user to use--username stringUsername for basic authentication to the API server--version version[=true]--version, --version=raw prints version information and quits; --version=vX.Y.Z... sets the reported version--warnings-as-errorsTreat warnings received from the server as errors and exit with a non-zero exit codeSee Alsokubectl - kubectl controls the Kubernetes cluster managerThis page is automatically generated.If you plan to report an issue with this page, mention that the page is auto-generated in your issue description. The fix may need to happen elsewhere in the Kubernetes project.


# Pod Overhead

Pod OverheadFEATURE STATE: Kubernetes v1.24 [stable]When you run a Pod on a Node, the Pod itself takes an amount of system resources. These resources are additional to the resources needed to run the container(s) inside the Pod. In Kubernetes, Pod Overhead is a way to account for the resources consumed by the Pod infrastructure on top of the container requests & limits.In Kubernetes, the Pod's overhead is set at admission time according to the overhead associated with the Pod's RuntimeClass.A pod's overhead is considered in addition to the sum of container resource requests when scheduling a Pod. Similarly, the kubelet will include the Pod overhead when sizing the Pod cgroup, and when carrying out Pod eviction ranking.Configuring Pod overheadYou need to make sure a RuntimeClass is utilized which defines the overhead field.Usage exampleTo work with Pod overhead, you need a RuntimeClass that defines the overhead field. As an example, you could use the following RuntimeClass definition with a virtualization container runtime (in this example, Kata Containers combined with the Firecracker virtual machine monitor) that uses around 120MiB per Pod for the virtual machine and the guest OS:# You need to change this example to match the actual runtime name, and per-Pod # resource overhead, that the container runtime is adding in your cluster. apiVersion: node.k8s.io/v1 kind: RuntimeClass metadata: name: kata-fc handler: kata-fc overhead: podFixed: memory: "120Mi" cpu: "250m" Workloads which are created which specify the kata-fc RuntimeClass handler will take the memory and cpu overheads into account for resource quota calculations, node scheduling, as well as Pod cgroup sizing.Consider running the given example workload, test-pod:apiVersion: v1 kind: Pod metadata: name: test-pod spec: runtimeClassName: kata-fc containers: - name: busybox-ctr image: busybox:1.28 stdin: true tty: true resources: limits: cpu: 500m memory: 100Mi - name: nginx-ctr image: nginx resources: limits: cpu: 1500m memory: 100Mi Note:If only limits are specified in the pod definition, kubelet will deduce requests from those limits and set them to be the same as the defined limits.At admission time the RuntimeClass admission controller updates the workload's PodSpec to include the overhead as described in the RuntimeClass. If the PodSpec already has this field defined, the Pod will be rejected. In the given example, since only the RuntimeClass name is specified, the admission controller mutates the Pod to include an overhead.After the RuntimeClass admission controller has made modifications, you can check the updated Pod overhead value:kubectl get pod test-pod -o jsonpath='{.spec.overhead}' The output is:map[cpu:250m memory:120Mi] If a ResourceQuota is defined, the sum of container requests as well as the overhead field are counted.When the kube-scheduler is deciding which node should run a new Pod, the scheduler considers that Pod's overhead as well as the sum of container requests for that Pod. For this example, the scheduler adds the requests and the overhead, then looks for a node that has 2.25 CPU and 320 MiB of memory available.Once a Pod is scheduled to a node, the kubelet on that node creates a new cgroup for the Pod. It is within this pod that the underlying container runtime will create containers.If the resource has a limit defined for each container (Guaranteed QoS or Burstable QoS with limits defined), the kubelet will set an upper limit for the pod cgroup associated with that resource (cpu.cfs_quota_us for CPU and memory.limit_in_bytes memory). This upper limit is based on the sum of the container limits plus the overhead defined in the PodSpec.For CPU, if the Pod is Guaranteed or Burstable QoS, the kubelet will set cpu.shares based on the sum of container requests plus the overhead defined in the PodSpec.Looking at our example, verify the container requests for the workload:kubectl get pod test-pod -o jsonpath='{.spec.containers[*].resources.limits}' The total container requests are 2000m CPU and 200MiB of memory:map[cpu: 500m memory:100Mi] map[cpu:1500m memory:100Mi] Check this against what is observed by the node:kubectl describe node | grep test-pod -B2 The output shows requests for 2250m CPU, and for 320MiB of memory. The requests include Pod overhead: Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits AGE --------- ---- ------------ ---------- --------------- ------------- --- default test-pod 2250m (56%) 2250m (56%) 320Mi (1%) 320Mi (1%) 36m Verify Pod cgroup limitsCheck the Pod's memory cgroups on the node where the workload is running. In the following example, crictl is used on the node, which provides a CLI for CRI-compatible container runtimes. This is an advanced example to show Pod overhead behavior, and it is not expected that users should need to check cgroups directly on the node.First, on the particular node, determine the Pod identifier:# Run this on the node where the Pod is scheduled POD_ID="$(sudo crictl pods --name test-pod -q)" From this, you can determine the cgroup path for the Pod:# Run this on the node where the Pod is scheduled sudo crictl inspectp -o=json $POD_ID | grep cgroupsPath The resulting cgroup path includes the Pod's pause container. The Pod level cgroup is one directory above. "cgroupsPath": "/kubepods/podd7f4b509-cf94-4951-9417-d1087c92a5b2/7ccf55aee35dd16aca4189c952d83487297f3cd760f1bbf09620e206e7d0c27a" In this specific case, the pod cgroup path is kubepods/podd7f4b509-cf94-4951-9417-d1087c92a5b2. Verify the Pod level cgroup setting for memory:# Run this on the node where the Pod is scheduled. # Also, change the name of the cgroup to match the cgroup allocated for your pod. cat /sys/fs/cgroup/memory/kubepods/podd7f4b509-cf94-4951-9417-d1087c92a5b2/memory.limit_in_bytes This is 320 MiB, as expected:335544320 ObservabilitySome kube_pod_overhead_* metrics are available in kube-state-metrics to help identify when Pod overhead is being utilized and to help observe stability of workloads running with a defined overhead.What's nextLearn more about RuntimeClassRead the PodOverhead Design enhancement proposal for extra context


# Restrict a Container's Syscalls with seccomp

Restrict a Container's Syscalls with seccompFEATURE STATE: Kubernetes v1.19 [stable]Seccomp stands for secure computing mode and has been a feature of the Linux kernel since version 2.6.12. It can be used to sandbox the privileges of a process, restricting the calls it is able to make from userspace into the kernel. Kubernetes lets you automatically apply seccomp profiles loaded onto a node to your Pods and containers.Identifying the privileges required for your workloads can be difficult. In this tutorial, you will go through how to load seccomp profiles into a local Kubernetes cluster, how to apply them to a Pod, and how you can begin to craft profiles that give only the necessary privileges to your container processes.ObjectivesLearn how to load seccomp profiles on a nodeLearn how to apply a seccomp profile to a containerObserve auditing of syscalls made by a container processObserve behavior when a missing profile is specifiedObserve a violation of a seccomp profileLearn how to create fine-grained seccomp profilesLearn how to apply a container runtime default seccomp profileBefore you beginIn order to complete all steps in this tutorial, you must install kind and kubectl.The commands used in the tutorial assume that you are using Docker as your container runtime. (The cluster that kind creates may use a different container runtime internally). You could also use Podman but in that case, you would have to follow specific instructions in order to complete the tasks successfully.This tutorial shows some examples that are still beta (since v1.25) and others that use only generally available seccomp functionality. You should make sure that your cluster is configured correctly for the version you are using.The tutorial also uses the curl tool for downloading examples to your computer. You can adapt the steps to use a different tool if you prefer.Note:It is not possible to apply a seccomp profile to a container running with privileged: true set in the container's securityContext. Privileged containers always run as Unconfined.Download example seccomp profilesThe contents of these profiles will be explored later on, but for now go ahead and download them into a directory named profiles/ so that they can be loaded into the cluster.pods/security/seccomp/profiles/audit.json { "defaultAction": "SCMP_ACT_LOG" }pods/security/seccomp/profiles/violation.json { "defaultAction": "SCMP_ACT_ERRNO" }pods/security/seccomp/profiles/fine-grained.json { "defaultAction": "SCMP_ACT_ERRNO", "architectures": [ "SCMP_ARCH_X86_64", "SCMP_ARCH_X86", "SCMP_ARCH_X32" ], "syscalls": [ { "names": [ "accept4", "epoll_wait", "pselect6", "futex", "madvise", "epoll_ctl", "getsockname", "setsockopt", "vfork", "mmap", "read", "write", "close", "arch_prctl", "sched_getaffinity", "munmap", "brk", "rt_sigaction", "rt_sigprocmask", "sigaltstack", "gettid", "clone", "bind", "socket", "openat", "readlinkat", "exit_group", "epoll_create1", "listen", "rt_sigreturn", "sched_yield", "clock_gettime", "connect", "dup2", "epoll_pwait", "execve", "exit", "fcntl", "getpid", "getuid", "ioctl", "mprotect", "nanosleep", "open", "poll", "recvfrom", "sendto", "set_tid_address", "setitimer", "writev", "fstatfs", "getdents64", "pipe2", "getrlimit" ], "action": "SCMP_ACT_ALLOW" } ] }Run these commands:mkdir ./profiles curl -L -o profiles/audit.json https://k8s.io/examples/pods/security/seccomp/profiles/audit.json curl -L -o profiles/violation.json https://k8s.io/examples/pods/security/seccomp/profiles/violation.json curl -L -o profiles/fine-grained.json https://k8s.io/examples/pods/security/seccomp/profiles/fine-grained.json ls profiles You should see three profiles listed at the end of the final step:audit.json fine-grained.json violation.json Create a local Kubernetes cluster with kindFor simplicity, kind can be used to create a single node cluster with the seccomp profiles loaded. Kind runs Kubernetes in Docker, so each node of the cluster is a container. This allows for files to be mounted in the filesystem of each container similar to loading files onto a node.pods/security/seccomp/kind.yaml apiVersion: kind.x-k8s.io/v1alpha4 kind: Cluster nodes: - role: control-plane extraMounts: - hostPath: "./profiles" containerPath: "/var/lib/kubelet/seccomp/profiles"Download that example kind configuration, and save it to a file named kind.yaml:curl -L -O https://k8s.io/examples/pods/security/seccomp/kind.yaml You can set a specific Kubernetes version by setting the node's container image. See Nodes within the kind documentation about configuration for more details on this. This tutorial assumes you are using Kubernetes v1.34.As a beta feature, you can configure Kubernetes to use the profile that the container runtime prefers by default, rather than falling back to Unconfined. If you want to try that, see enable the use of RuntimeDefault as the default seccomp profile for all workloads before you continue.Once you have a kind configuration in place, create the kind cluster with that configuration:kind create cluster --config=kind.yaml After the new Kubernetes cluster is ready, identify the Docker container running as the single node cluster:docker ps You should see output indicating that a container is running with name kind-control-plane. The output is similar to:CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 6a96207fed4b kindest/node:v1.18.2 "/usr/local/bin/entr…" 27 seconds ago Up 24 seconds 127.0.0.1:42223->6443/tcp kind-control-plane If observing the filesystem of that container, you should see that the profiles/ directory has been successfully loaded into the default seccomp path of the kubelet. Use docker exec to run a command in the Pod:# Change 6a96207fed4b to the container ID you saw from "docker ps" docker exec -it 6a96207fed4b ls /var/lib/kubelet/seccomp/profiles audit.json fine-grained.json violation.json You have verified that these seccomp profiles are available to the kubelet running within kind.Create a Pod that uses the container runtime default seccomp profileMost container runtimes provide a sane set of default syscalls that are allowed or not. You can adopt these defaults for your workload by setting the seccomp type in the security context of a pod or container to RuntimeDefault.Note:If you have the seccompDefault configuration enabled, then Pods use the RuntimeDefault seccomp profile whenever no other seccomp profile is specified. Otherwise, the default is Unconfined.Here's a manifest for a Pod that requests the RuntimeDefault seccomp profile for all its containers:pods/security/seccomp/ga/default-pod.yaml apiVersion: v1 kind: Pod metadata: name: default-pod labels: app: default-pod spec: securityContext: seccompProfile: type: RuntimeDefault containers: - name: test-container image: hashicorp/http-echo:1.0 args: - "-text=just made some more syscalls!" securityContext: allowPrivilegeEscalation: falseCreate that Pod:kubectl apply -f https://k8s.io/examples/pods/security/seccomp/ga/default-pod.yaml kubectl get pod default-pod The Pod should be showing as having started successfully:NAME READY STATUS RESTARTS AGE default-pod 1/1 Running 0 20s Delete the Pod before moving to the