# Admission Control in Kubernetes

Admission Control in KubernetesThis page provides an overview of admission controllers.An admission controller is a piece of code that intercepts requests to the Kubernetes API server prior to persistence of the resource, but after the request is authenticated and authorized.Several important features of Kubernetes require an admission controller to be enabled in order to properly support the feature. As a result, a Kubernetes API server that is not properly configured with the right set of admission controllers is an incomplete server that will not support all the features you expect.What are they?Admission controllers are code within the Kubernetes API server that check the data arriving in a request to modify a resource.Admission controllers apply to requests that create, delete, or modify objects. Admission controllers can also block custom verbs, such as a request to connect to a pod via an API server proxy. Admission controllers do not (and cannot) block requests to read (get, watch or list) objects, because reads bypass the admission control layer.Admission control mechanisms may be validating, mutating, or both. Mutating controllers may modify the data for the resource being modified; validating controllers may not.The admission controllers in Kubernetes 1.34 consist of the list below, are compiled into the kube-apiserver binary, and may only be configured by the cluster administrator.Admission control extension pointsWithin the full list, there are three special controllers: MutatingAdmissionWebhook, ValidatingAdmissionWebhook, and ValidatingAdmissionPolicy. The two webhook controllers execute the mutating and validating (respectively) admission control webhooks which are configured in the API. ValidatingAdmissionPolicy provides a way to embed declarative validation code within the API, without relying on any external HTTP callouts.You can use these three admission controllers to customize cluster behavior at admission time.Admission control phasesThe admission control process proceeds in two phases. In the first phase, mutating admission controllers are run. In the second phase, validating admission controllers are run. Note again that some of the controllers are both.If any of the controllers in either phase reject the request, the entire request is rejected immediately and an error is returned to the end-user.Finally, in addition to sometimes mutating the object in question, admission controllers may sometimes have side effects, that is, mutate related resources as part of request processing. Incrementing quota usage is the canonical example of why this is necessary. Any such side-effect needs a corresponding reclamation or reconciliation process, as a given admission controller does not know for sure that a given request will pass all of the other admission controllers.The ordering of these calls can be seen below.Why do I need them?Several important features of Kubernetes require an admission controller to be enabled in order to properly support the feature. As a result, a Kubernetes API server that is not properly configured with the right set of admission controllers is an incomplete server and will not support all the features you expect.How do I turn on an admission controller?The Kubernetes API server flag enable-admission-plugins takes a comma-delimited list of admission control plugins to invoke prior to modifying objects in the cluster. For example, the following command line enables the NamespaceLifecycle and the LimitRanger admission control plugins:kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger ... Note:Depending on the way your Kubernetes cluster is deployed and how the API server is started, you may need to apply the settings in different ways. For example, you may have to modify the systemd unit file if the API server is deployed as a systemd service, you may modify the manifest file for the API server if Kubernetes is deployed in a self-hosted way.How do I turn off an admission controller?The Kubernetes API server flag disable-admission-plugins takes a comma-delimited list of admission control plugins to be disabled, even if they are in the list of plugins enabled by default.kube-apiserver --disable-admission-plugins=PodNodeSelector,AlwaysDeny ... Which plugins are enabled by default?To see which admission plugins are enabled:kube-apiserver -h | grep enable-admission-plugins In Kubernetes 1.34, the default ones are:CertificateApproval, CertificateSigning, CertificateSubjectRestriction, DefaultIngressClass, DefaultStorageClass, DefaultTolerationSeconds, LimitRanger, MutatingAdmissionWebhook, NamespaceLifecycle, PersistentVolumeClaimResize, PodSecurity, Priority, ResourceQuota, RuntimeClass, ServiceAccount, StorageObjectInUseProtection, TaintNodesByCondition, ValidatingAdmissionPolicy, ValidatingAdmissionWebhook What does each admission controller do?AlwaysAdmitFEATURE STATE: Kubernetes v1.13 [deprecated]Type: Validating.This admission controller allows all pods into the cluster. It is deprecated because its behavior is the same as if there were no admission controller at all.AlwaysDenyFEATURE STATE: Kubernetes v1.13 [deprecated]Type: Validating.Rejects all requests. AlwaysDeny is deprecated as it has no real meaning.AlwaysPullImagesType: Mutating and Validating.This admission controller modifies every new Pod to force the image pull policy to Always. This is useful in a multitenant cluster so that users can be assured that their private images can only be used by those who have the credentials to pull them. Without this admission controller, once an image has been pulled to a node, any pod from any user can use it by knowing the image's name (assuming the Pod is scheduled onto the right node), without any authorization check against the image. When this admission controller is enabled, images are always pulled prior to starting containers, which means valid credentials are required.CertificateApprovalType: Validating.This admission controller observes requests to approve CertificateSigningRequest resources and performs additional authorization checks to ensure the approving user has permission to approve certificate requests with the spec.signerName requested on the CertificateSigningRequest resource.See Certificate Signing Requests for more information on the permissions required to perform different actions on CertificateSigningRequest resources.CertificateSigningType: Validating.This admission controller observes updates to the status.certificate field of CertificateSigningRequest resources and performs an additional authorization checks to ensure the signing user has permission to sign certificate requests with the spec.signerName requested on the CertificateSigningRequest resource.See Certificate Signing Requests for more information on the permissions required to perform different actions on CertificateSigningRequest resources.CertificateSubjectRestrictionType: Validating.This admission controller observes creation of CertificateSigningRequest resources that have a spec.signerName of kubernetes.io/kube-apiserver-client. It rejects any request that specifies a 'group' (or 'organization attribute') of system:masters.DefaultIngressClassType: Mutating.This admission controller observes creation of Ingress objects that do not request any specific ingress class and automatically adds a default ingress class to them. This way, users that do not request any special ingress class do not need to care about them at all and they will get the default one.This admission controller does not do anything when no default ingress class is configured. When more than one ingress class is marked as default, it rejects any creation of Ingress with an error and an administrator must revisit their IngressClass objects and mark only one as default (with the annotation "ingressclass.kubernetes.io/is-default-class"). This admission controller ignores any Ingress updates; it acts only on creation.See the Ingress documentation for more about ingress classes and how to mark one as default.DefaultStorageClassType: Mutating.This admission controller observes creation of PersistentVolumeClaim objects that do not request any specific storage class and automatically adds a default storage class to them. This way, users that do not request any special storage class do not need to care about them at all and they will get the default one.This admission controller does nothing when no default StorageClass exists. When more than one storage class is marked as default, and you then create a PersistentVolumeClaim with no storageClassName set, Kubernetes uses the most recently created default StorageClass. When a PersistentVolumeClaim is created with a specified volumeName, it remains in a pending state if the static volume's storageClassName does not match the storageClassName on the PersistentVolumeClaim after any default StorageClass is applied to it. This admission controller ignores any PersistentVolumeClaim updates; it acts only on creation.See persistent volume documentation about persistent volume claims and storage classes and how to mark a storage class as default.DefaultTolerationSecondsType: Mutating.This admission controller sets the default forgiveness toleration for pods to tolerate the taints notready:NoExecute and unreachable:NoExecute based on the k8s-apiserver input parameters default-not-ready-toleration-seconds and default-unreachable-toleration-seconds if the pods don't already have toleration for taints node.kubernetes.io/not-ready:NoExecute or node.kubernetes.io/unreachable:NoExecute. The default value for default-not-ready-toleration-seconds and default-unreachable-toleration-seconds is 5 minutes.DenyServiceExternalIPsType: Validating.This admission controller rejects all net-new usage of the Service field externalIPs. This feature is very powerful (allows network traffic interception) and not well controlled by policy. When enabled, users of the cluster may not create new Services which use externalIPs and may not add new values to externalIPs on existing Service objects. Existing uses of externalIPs are not affected, and users may remove values from externalIPs on existing Service objects.Most users do not need this feature at all, and cluster admins should consider disabling it. Clusters that do need to use this feature should consider using some custom policy to manage usage of it.This admission controller is disabled by default.EventRateLimitFEATURE STATE: Kubernetes v1.13 [alpha]Type: Validating.This admission controller mitigates the problem where the API server gets flooded by requests to store new Events. The cluster admin can specify event rate limits by:Enabling the EventRateLimit admission controller;Referencing an EventRateLimit configuration file from the file provided to the API server's command line flag --admission-control-config-file:apiVersion: apiserver.config.k8s.io/v1 kind: AdmissionConfiguration plugins: - name: EventRateLimit path: eventconfig.yaml ... There are four types of limits that can be specified in the configuration:Server: All Event requests (creation or modifications) received by the API server share a single bucket.Namespace: Each namespace has a dedicated bucket.User: Each user is allocated a bucket.SourceAndObject: A bucket is assigned by each combination of source and involved object of the event.Below is a sample eventconfig.yaml for such a configuration:apiVersion: eventratelimit.admission.k8s.io/v1alpha1 kind: Configuration limits: - type: Namespace qps: 50 burst: 100 cacheSize: 2000 - type: User qps: 10 burst: 50 See the EventRateLimit Config API (v1alpha1) for more details.This admission controller is disabled by default.ExtendedResourceTolerationType: Mutating.This plug-in facilitates creation of dedicated nodes with extended resources. If operators want to create dedicated nodes with extended resources (like GPUs, FPGAs etc.), they are expected to taint the node with the extended resource name as the key. This admission controller, if enabled, automatically adds tolerations for such taints to pods requesting extended resources, so users don't have to manually add these tolerations.This admission controller is disabled by default.ImagePolicyWebhookType: Validating.The ImagePolicyWebhook admission controller allows a backend webhook to make admission decisions.This admission controller is disabled by default.Configuration file formatImagePolicyWebhook uses a configuration file to set options for the behavior of the backend. This file may be json or yaml and has the following format:imagePolicy: kubeConfigFile: /path/to/kubeconfig/for/backend # time in s to cache approval allowTTL: 50 # time in s to cache denial denyTTL: 50 # time in ms to wait between retries retryBackoff: 500 # determines behavior if the webhook backend fails defaultAllow: true Reference the ImagePolicyWebhook configuration file from the file provided to the API server's command line flag --admission-control-config-file:apiVersion: apiserver.config.k8s.io/v1 kind: AdmissionConfiguration plugins: - name: ImagePolicyWebhook path: imagepolicyconfig.yaml ... Alternatively, you can embed the configuration directly in the file:apiVersion: apiserver.config.k8s.io/v1 kind: AdmissionConfiguration plugins: - name: ImagePolicyWebhook configuration: imagePolicy: kubeConfigFile: <path-to-kubeconfig-file> allowTTL: 50 denyTTL: 50 retryBackoff: 500 defaultAllow: true The ImagePolicyWebhook config file must reference a kubeconfig formatted file which sets up the connection to the backend. It is required that the backend communicate over TLS.The kubeconfig file's cluster field must point to the remote service, and the user field must contain the returned authorizer.# clusters refers to the remote service. clusters: - name: name-of-remote-imagepolicy-service cluster: certificate-authority: /path/to/ca.pem # CA for verifying the remote service. server: https://images.example.com/policy # URL of remote service to query. Must use 'https'. # users refers to the API server's webhook configuration. users: - name: name-of-api-server user: client-certificate: /path/to/cert.pem # cert for the webhook admission controller to use client-key: /path/to/key.pem # key matching the cert For additional HTTP configuration, refer to the kubeconfig documentation.Request payloadsWhen faced with an admission decision, the API Server POSTs a JSON serialized imagepolicy.k8s.io/v1alpha1 ImageReview object describing the action. This object contains fields describing the containers being admitted, as well as any pod annotations that match *.image-policy.k8s.io/*.Note:The webhook API objects are subject to the same versioning compatibility rules as other Kubernetes API objects. Implementers should be aware of looser compatibility promises for alpha objects and check the apiVersion field of the request to ensure correct deserialization. Additionally, the API Server must enable the imagepolicy.k8s.io/v1alpha1 API extensions group (--runtime-config=imagepolicy.k8s.io/v1alpha1=true).An example request body:{ "apiVersion": "imagepolicy.k8s.io/v1alpha1", "kind": "ImageReview", "spec": { "containers": [ { "image": "myrepo/myimage:v1" }, { "image": "myrepo/myimage@sha256:beb6bd6a68f114c1dc2ea4b28db81bdf91de202a9014972bec5e4d9171d90ed" } ], "annotations": { "mycluster.image-policy.k8s.io/ticket-1234": "break-glass" }, "namespace": "mynamespace" } } The remote service is expected to fill the status field of the request and respond to either allow or disallow access. The response body's spec field is ignored, and may be omitted. A permissive response would return:{ "apiVersion": "imagepolicy.k8s.io/v1alpha1", "kind": "ImageReview", "status": { "allowed": true } } To disallow access, the service would return:{ "apiVersion": "imagepolicy.k8s.io/v1alpha1", "kind": "ImageReview", "status": { "allowed": false, "reason": "image currently blacklisted" } } For further documentation refer to the imagepolicy.v1alpha1 API.Extending with AnnotationsAll annotations on a Pod that match *.image-policy.k8s.io/* are sent to the webhook. Sending annotations allows users who are aware of the image policy backend to send extra information to it, and for different backends implementations to accept different information.Examples of information you might put here are:request to "break glass" to override a policy, in case of emergency.a ticket number from a ticket system that documents the break-glass requestprovide a hint to the policy server as to the imageID of the image being provided, to save it a lookupIn any case, the annotations are provided by the user and are not validated by Kubernetes in any way.LimitPodHardAntiAffinityTopologyType: Validating.This admission controller denies any pod that defines AntiAffinity topology key other than kubernetes.io/hostname in requiredDuringSchedulingRequiredDuringExecution.This admission controller is disabled by default.LimitRangerType: Mutating and Validating.This admission controller will observe the incoming request and ensure that it does not violate any of the constraints enumerated in the LimitRange object in a Namespace. If you are using LimitRange objects in your Kubernetes deployment, you MUST use this admission controller to enforce those constraints. LimitRanger can also be used to apply default resource requests to Pods that don't specify any; currently, the default LimitRanger applies a 0.1 CPU requirement to all Pods in the default namespace.See the LimitRange API reference and the example of LimitRange for more details.MutatingAdmissionWebhookType: Mutating.This admission controller calls any mutating webhooks which match the request. Matching webhooks are called in serial; each one may modify the object if it desires.This admission controller (as implied by the name) only runs in the mutating phase.If a webhook called by this has side effects (for example, decrementing quota) it must have a reconciliation system, as it is not guaranteed that subsequent webhooks or validating admission controllers will permit the request to finish.If you disable the MutatingAdmissionWebhook, you must also disable the MutatingWebhookConfiguration object in the admissionregistration.k8s.io/v1 group/version via the --runtime-config flag, both are on by default.Use caution when authoring and installing mutating webhooksUsers may be confused when the objects they try to create are different from what they get back.Built in control loops may break when the objects they try to create are different when read back.Setting originally unset fields is less likely to cause problems than overwriting fields set in the original request. Avoid doing the latter.Future changes to control loops for built-in resources or third-party resources may break webhooks that work well today. Even when the webhook installation API is finalized, not all possible webhook behaviors will be guaranteed to be supported indefinitely.NamespaceAutoProvisionType: Mutating.This admission controller examines all incoming requests on namespaced resources and checks if the referenced namespace does exist. It creates a namespace if it cannot be found. This admission controller is useful in deployments that do not want to restrict creation of a namespace prior to its usage.NamespaceExistsType: Validating.This admission controller checks all requests on namespaced resources other than Namespace itself. If the namespace referenced from a request doesn't exist, the request is rejected.NamespaceLifecycleType: Validating.This admission controller enforces that a Namespace that is undergoing termination cannot have new objects created in it, and ensures that requests in a non-existent Namespace are rejected. This admission controller also prevents deletion of three system reserved namespaces default, kube-system, kube-public.A Namespace deletion kicks off a sequence of operations that remove all objects (pods, services, etc.) in that namespace. In order to enforce integrity of that process, we strongly recommend running this admission controller.NodeRestrictionType: Validating.This admission controller limits the Node and Pod objects a kubelet can modify. In order to be limited by this admission controller, kubelets must use credentials in the system:nodes group, with a username in the form system:node:<nodeName>. Such kubelets will only be allowed to modify their own Node API object, and only modify Pod API objects that are bound to their node. kubelets are not allowed to update or remove taints from their Node API object.The NodeRestriction admission plugin prevents kubelets from deleting their Node API object, and enforces kubelet modification of labels under the kubernetes.io/ or k8s.io/ prefixes as follows:Prevents kubelets from adding/removing/updating labels with a node-restriction.kubernetes.io/ prefix. This label prefix is reserved for administrators to label their Node objects for workload isolation purposes, and kubelets will not be allowed to modify labels with that prefix.Allows kubelets to add/remove/update these labels and label prefixes:kubernetes.io/hostnamekubernetes.io/archkubernetes.io/osbeta.kubernetes.io/instance-typenode.kubernetes.io/instance-typefailure-domain.beta.kubernetes.io/region (deprecated)failure-domain.beta.kubernetes.io/zone (deprecated)topology.kubernetes.io/regiontopology.kubernetes.io/zonekubelet.kubernetes.io/-prefixed labelsnode.kubernetes.io/-prefixed labelsUse of any other labels under the kubernetes.io or k8s.io prefixes by kubelets is reserved, and may be disallowed or allowed by the NodeRestriction admission plugin in the future.Future versions may add additional restrictions to ensure kubelets have the minimal set of permissions required to operate correctly.OwnerReferencesPermissionEnforcementType: Validating.This admission controller protects the access to the metadata.ownerReferences of an object so that only users with delete permission to the object can change it. This admission controller also protects the access to metadata.ownerReferences[x].blockOwnerDeletion of an object, so that only users with update permission to the finalizers subresource of the referenced owner can change it.PersistentVolumeClaimResizeFEATURE STATE: Kubernetes v1.24 [stable]Type: Validating.This admission controller implements additional validations for checking incoming PersistentVolumeClaim resize requests.Enabling the PersistentVolumeClaimResize admission controller is recommended. This admission controller prevents resizing of all claims by default unless a claim's StorageClass explicitly enables resizing by setting allowVolumeExpansion to true.For example: all PersistentVolumeClaims created from the following StorageClass support volume expansion:apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: gluster-vol-default provisioner: kubernetes.io/glusterfs parameters: resturl: "http://192.168.10.100:8080" restuser: "" secretNamespace: "" secretName: "" allowVolumeExpansion: true For more information about persistent volume claims, see PersistentVolumeClaims.PodNodeSelectorFEATURE STATE: Kubernetes v1.5 [alpha]Type: Validating.This admission controller defaults and limits what node selectors may be used within a namespace by reading a namespace annotation and a global configuration.This admission controller is disabled by default.Configuration file formatPodNodeSelector uses a configuration file to set options for the behavior of the backend. Note that the configuration file format will move to a versioned file in a future release. This file may be json or yaml and has the following format:podNodeSelectorPluginConfig: clusterDefaultNodeSelector: name-of-node-selector namespace1: name-of-node-selector namespace2: name-of-node-selector Reference the PodNodeSelector configuration file from the file provided to the API server's command line flag --admission-control-config-file:apiVersion: apiserver.config.k8s.io/v1 kind: AdmissionConfiguration plugins: - name: PodNodeSelector path: podnodeselector.yaml ... Configuration Annotation FormatPodNodeSelector uses the annotation key scheduler.alpha.kubernetes.io/node-selector to assign node selectors to namespaces.apiVersion: v1 kind: Namespace metadata: annotations: scheduler.alpha.kubernetes.io/node-selector: name-of-node-selector name: namespace3 Internal BehaviorThis admission controller has the following behavior:If the Namespace has an annotation with a key scheduler.alpha.kubernetes.io/node-selector, use its value as the node selector.If the namespace lacks such an annotation, use the clusterDefaultNodeSelector defined in the PodNodeSelector plugin configuration file as the node selector.Evaluate the pod's node selector against the namespace node selector for conflicts. Conflicts result in rejection.Evaluate the pod's node selector against the namespace-specific allowed selector defined the plugin configuration file. Conflicts result in rejection.Note:PodNodeSelector allows forcing pods to run on specifically labeled nodes. Also see the PodTolerationRestriction admission plugin, which allows preventing pods from running on specifically tainted nodes.PodSecurityFEATURE STATE: Kubernetes v1.25 [stable]Type: Validating.The PodSecurity admission controller checks new Pods before they are admitted, determines if it should be admitted based on the requested security context and the restrictions on permitted Pod Security Standards for the namespace that the Pod would be in.See the Pod Security Admission documentation for more information.PodSecurity replaced an older admission controller named PodSecurityPolicy.PodTolerationRestrictionFEATURE STATE: Kubernetes v1.7 [alpha]Type: Mutating and Validating.The PodTolerationRestriction admission controller verifies any conflict between tolerations of a pod and the tolerations of its namespace. It rejects the pod request if there is a conflict. It then merges the tolerations annotated on the namespace into the tolerations of the pod. The resulting tolerations are checked against a list of allowed tolerations annotated to the namespace. If the check succeeds, the pod request is admitted otherwise it is rejected.If the namespace of the pod does not have any associated default tolerations or allowed tolerations annotated, the cluster-level default tolerations or cluster-level list of allowed tolerations are used instead if they are specified.Tolerations to a namespace are assigned via the scheduler.alpha.kubernetes.io/defaultTolerations annotation key. The list of allowed tolerations can be added via the scheduler.alpha.kubernetes.io/tolerationsWhitelist annotation key.Example for namespace annotations:apiVersion: v1 kind: Namespace metadata: name: apps-that-need-nodes-exclusively annotations: scheduler.alpha.kubernetes.io/defaultTolerations: '[{"operator": "Exists", "effect": "NoSchedule", "key": "dedicated-node"}]' scheduler.alpha.kubernetes.io/tolerationsWhitelist: '[{"operator": "Exists", "effect": "NoSchedule", "key": "dedicated-node"}]' This admission controller is disabled by default.PodTopologyLabelsFEATURE STATE: Kubernetes v1.34 []Type: MutatingThe PodTopologyLabels admission controller mutates the pods/binding subresources for all pods bound to a Node, adding topology labels matching those of the bound Node. This allows Node topology labels to be available as pod labels, which can be surfaced to running containers using the Downward API. The labels available as a result of this controller are the topology.kubernetes.io/region and topology.kuberentes.io/zone labels.Note:If any mutating admission webhook adds or modifies labels of the pods/binding subresource, these changes will propagate to pod labels as a result of this controller, overwriting labels with conflicting keys.This admission controller is enabled when the PodTopologyLabelsAdmission feature gate is enabled.PriorityType: Mutating and Validating.The priority admission controller uses the priorityClassName field and populates the integer value of the priority. If the priority class is not found, the Pod is rejected.ResourceQuotaType: Validating.This admission controller will observe the incoming request and ensure that it does not violate any of the constraints enumerated in the ResourceQuota object in a Namespace. If you are using ResourceQuota objects in your Kubernetes deployment, you MUST use this admission controller to enforce quota constraints.See the ResourceQuota API reference and the example of Resource Quota for more details.RuntimeClassType: Mutating and Validating.If you define a RuntimeClass with Pod overhead configured, this admission controller checks incoming Pods. When enabled, this admission controller rejects any Pod create requests that have the overhead already set. For Pods that have a RuntimeClass configured and selected in their .spec, this admission controller sets .spec.overhead in the Pod based on the value defined in the corresponding RuntimeClass.See also Pod Overhead for more information.ServiceAccountType: Mutating and Validating.This admission controller implements automation for serviceAccounts. The Kubernetes project strongly recommends enabling this admission controller. You should enable this admission controller if you intend to make any use of Kubernetes ServiceAccount objects.To enhance the security measures around Secrets, use separate namespaces to isolate access to mounted secrets.StorageObjectInUseProtectionType: Mutating.The StorageObjectInUseProtection plugin adds the kubernetes.io/pvc-protection or kubernetes.io/pv-protection finalizers to newly created Persistent Volume Claims (PVCs) or Persistent Volumes (PV). In case a user deletes a PVC or PV the PVC or PV is not removed until the finalizer is removed from the PVC or PV by PVC or PV Protection Controller. Refer to the Storage Object in Use Protection for more detailed information.TaintNodesByConditionType: Mutating.This admission controller taints newly created Nodes as NotReady and NoSchedule. That tainting avoids a race condition that could cause Pods to be scheduled on new Nodes before their taints were updated to accurately reflect their reported conditions.ValidatingAdmissionPolicyType: Validating.This admission controller implements the CEL validation for incoming matched requests. It is enabled when both feature gate validatingadmissionpolicy and admissionregistration.k8s.io/v1alpha1 group/version are enabled. If any of the ValidatingAdmissionPolicy fails, the request fails.ValidatingAdmissionWebhookType: Validating.This admission controller calls any validating webhooks which match the request. Matching webhooks are called in parallel; if any of them rejects the request, the request fails. This admission controller only runs in the validation phase; the webhooks it calls may not mutate the object, as opposed to the webhooks called by the MutatingAdmissionWebhook admission controller.If a webhook called by this has side effects (for example, decrementing quota) it must have a reconciliation system, as it is not guaranteed that subsequent webhooks or other validating admission controllers will permit the request to finish.If you disable the ValidatingAdmissionWebhook, you must also disable the ValidatingWebhookConfiguration object in the admissionregistration.k8s.io/v1 group/version via the --runtime-config flag.Is there a recommended set of admission controllers to use?Yes. The recommended admission controllers are enabled by default (shown here), so you do not need to explicitly specify them. You can enable additional admission controllers beyond the default set using the --enable-admission-plugins flag (order doesn't matter).


# Running Automated Tasks with a CronJob

Running Automated Tasks with a CronJobThis page shows how to run automated tasks using Kubernetes CronJob object.Before you beginYou need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. It is recommended to run this tutorial on a cluster with at least two nodes that are not acting as control plane hosts. If you do not already have a cluster, you can create one by using minikube or you can use one of these Kubernetes playgrounds:iximiuz LabsKillercodaKodeKloudPlay with KubernetesCreating a CronJobCron jobs require a config file. Here is a manifest for a CronJob that runs a simple demonstration task every minute:application/job/cronjob.yaml apiVersion: batch/v1 kind: CronJob metadata: name: hello spec: schedule: "* * * * *" jobTemplate: spec: template: spec: containers: - name: hello image: busybox:1.28 imagePullPolicy: IfNotPresent command: - /bin/sh - -c - date; echo Hello from the Kubernetes cluster restartPolicy: OnFailure Run the example CronJob by using this command:kubectl create -f https://k8s.io/examples/application/job/cronjob.yaml The output is similar to this:cronjob.batch/hello created After creating the cron job, get its status using this command:kubectl get cronjob hello The output is similar to this:NAME SCHEDULE SUSPEND ACTIVE LAST SCHEDULE AGE hello */1 * * * * False 0 <none> 10s As you can see from the results of the command, the cron job has not scheduled or run any jobs yet. Watch for the job to be created in around one minute:kubectl get jobs --watch The output is similar to this:NAME COMPLETIONS DURATION AGE hello-4111706356 0/1 0s hello-4111706356 0/1 0s 0s hello-4111706356 1/1 5s 5s Now you've seen one running job scheduled by the "hello" cron job. You can stop watching the job and view the cron job again to see that it scheduled the job:kubectl get cronjob hello The output is similar to this:NAME SCHEDULE SUSPEND ACTIVE LAST SCHEDULE AGE hello */1 * * * * False 0 50s 75s You should see that the cron job hello successfully scheduled a job at the time specified in LAST SCHEDULE. There are currently 0 active jobs, meaning that the job has completed or failed.Now, find the pods that the last scheduled job created and view the standard output of one of the pods.Note:The job name is different from the pod name.# Replace "hello-4111706356" with the job name in your system pods=$(kubectl get pods --selector=job-name=hello-4111706356 --output=jsonpath={.items[*].metadata.name}) Show the pod log:kubectl logs $pods The output is similar to this:Fri Feb 22 11:02:09 UTC 2019 Hello from the Kubernetes cluster Deleting a CronJobWhen you don't need a cron job any more, delete it with kubectl delete cronjob <cronjob name>:kubectl delete cronjob hello Deleting the cron job removes all the jobs and pods it created and stops it from creating additional jobs. You can read more about removing jobs in garbage collection.


# Containers

ContainersTechnology for packaging an application along with its runtime dependencies.This page will discuss containers and container images, as well as their use in operations and solution development.The word container is an overloaded term. Whenever you use the word, check whether your audience uses the same definition.Each container that you run is repeatable; the standardization from having dependencies included means that you get the same behavior wherever you run it.Containers decouple applications from the underlying host infrastructure. This makes deployment easier in different cloud or OS environments.Each node in a Kubernetes cluster runs the containers that form the Pods assigned to that node. Containers in a Pod are co-located and co-scheduled to run on the same node.Container imagesA container image is a ready-to-run software package containing everything needed to run an application: the code and any runtime it requires, application and system libraries, and default values for any essential settings.Containers are intended to be stateless and immutable: you should not change the code of a container that is already running. If you have a containerized application and want to make changes, the correct process is to build a new image that includes the change, then recreate the container to start from the updated image.Container runtimesA fundamental component that empowers Kubernetes to run containers effectively. It is responsible for managing the execution and lifecycle of containers within the Kubernetes environment.Kubernetes supports container runtimes such as containerd, CRI-O, and any other implementation of the Kubernetes CRI (Container Runtime Interface).Usually, you can allow your cluster to pick the default container runtime for a Pod. If you need to use more than one container runtime in your cluster, you can specify the RuntimeClass for a Pod to make sure that Kubernetes runs those containers using a particular container runtime.You can also use RuntimeClass to run different Pods with the same container runtime but with different settings.Container EnvironmentContainer Lifecycle Hooks


# ConfigMap

ConfigMapConfigMap holds configuration data for pods to consume.apiVersion: v1import "k8s.io/api/core/v1"ConfigMapConfigMap holds configuration data for pods to consume.apiVersion: v1kind: ConfigMapmetadata (ObjectMeta)Standard object's metadata. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadatabinaryData (map[string][]byte)BinaryData contains the binary data. Each key must consist of alphanumeric characters, '-', '_' or '.'. BinaryData can contain byte sequences that are not in the UTF-8 range. The keys stored in BinaryData must not overlap with the ones in the Data field, this is enforced during validation process. Using this field will require 1.10+ apiserver and kubelet.data (map[string]string)Data contains the configuration data. Each key must consist of alphanumeric characters, '-', '_' or '.'. Values with non-UTF-8 byte sequences must use the BinaryData field. The keys stored in Data must not overlap with the keys in the BinaryData field, this is enforced during validation process.immutable (boolean)Immutable, if set to true, ensures that data stored in the ConfigMap cannot be updated (only object metadata can be modified). If not set to true, the field can be modified at any time. Defaulted to nil.ConfigMapListConfigMapList is a resource containing a list of ConfigMap objects.apiVersion: v1kind: ConfigMapListmetadata (ListMeta)More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadataitems ([]ConfigMap), requiredItems is the list of ConfigMaps.Operationsget read the specified ConfigMapHTTP RequestGET /api/v1/namespaces/{namespace}/configmaps/{name}Parametersname (in path): string, requiredname of the ConfigMapnamespace (in path): string, requirednamespacepretty (in query): stringprettyResponse200 (ConfigMap): OK401: Unauthorizedlist list or watch objects of kind ConfigMapHTTP RequestGET /api/v1/namespaces/{namespace}/configmapsParametersnamespace (in path): string, requirednamespaceallowWatchBookmarks (in query): booleanallowWatchBookmarkscontinue (in query): stringcontinuefieldSelector (in query): stringfieldSelectorlabelSelector (in query): stringlabelSelectorlimit (in query): integerlimitpretty (in query): stringprettyresourceVersion (in query): stringresourceVersionresourceVersionMatch (in query): stringresourceVersionMatchsendInitialEvents (in query): booleansendInitialEventstimeoutSeconds (in query): integertimeoutSecondswatch (in query): booleanwatchResponse200 (ConfigMapList): OK401: Unauthorizedlist list or watch objects of kind ConfigMapHTTP RequestGET /api/v1/configmapsParametersallowWatchBookmarks (in query): booleanallowWatchBookmarkscontinue (in query): stringcontinuefieldSelector (in query): stringfieldSelectorlabelSelector (in query): stringlabelSelectorlimit (in query): integerlimitpretty (in query): stringprettyresourceVersion (in query): stringresourceVersionresourceVersionMatch (in query): stringresourceVersionMatchsendInitialEvents (in query): booleansendInitialEventstimeoutSeconds (in query): integertimeoutSecondswatch (in query): booleanwatchResponse200 (ConfigMapList): OK401: Unauthorizedcreate create a ConfigMapHTTP RequestPOST /api/v1/namespaces/{namespace}/configmapsParametersnamespace (in path): string, requirednamespacebody: ConfigMap, requireddryRun (in query): stringdryRunfieldManager (in query): stringfieldManagerfieldValidation (in query): stringfieldValidationpretty (in query): stringprettyResponse200 (ConfigMap): OK201 (ConfigMap): Created202 (ConfigMap): Accepted401: Unauthorizedupdate replace the specified ConfigMapHTTP RequestPUT /api/v1/namespaces/{namespace}/configmaps/{name}Parametersname (in path): string, requiredname of the ConfigMapnamespace (in path): string, requirednamespacebody: ConfigMap, requireddryRun (in query): stringdryRunfieldManager (in query): stringfieldManagerfieldValidation (in query): stringfieldValidationpretty (in query): stringprettyResponse200 (ConfigMap): OK201 (ConfigMap): Created401: Unauthorizedpatch partially update the specified ConfigMapHTTP RequestPATCH /api/v1/namespaces/{namespace}/configmaps/{name}Parametersname (in path): string, requiredname of the ConfigMapnamespace (in path): string, requirednamespacebody: Patch, requireddryRun (in query): stringdryRunfieldManager (in query): stringfieldManagerfieldValidation (in query): stringfieldValidationforce (in query): booleanforcepretty (in query): stringprettyResponse200 (ConfigMap): OK201 (ConfigMap): Created401: Unauthorizeddelete delete a ConfigMapHTTP RequestDELETE /api/v1/namespaces/{namespace}/configmaps/{name}Parametersname (in path): string, requiredname of the ConfigMapnamespace (in path): string, requirednamespacebody: DeleteOptionsdryRun (in query): stringdryRungracePeriodSeconds (in query): integergracePeriodSecondsignoreStoreReadErrorWithClusterBreakingPotential (in query): booleanignoreStoreReadErrorWithClusterBreakingPotentialpretty (in query): stringprettypropagationPolicy (in query): stringpropagationPolicyResponse200 (Status): OK202 (Status): Accepted401: Unauthorizeddeletecollection delete collection of ConfigMapHTTP RequestDELETE /api/v1/namespaces/{namespace}/configmapsParametersnamespace (in path): string, requirednamespacebody: DeleteOptionscontinue (in query): stringcontinuedryRun (in query): stringdryRunfieldSelector (in query): stringfieldSelectorgracePeriodSeconds (in query): integergracePeriodSecondsignoreStoreReadErrorWithClusterBreakingPotential (in query): booleanignoreStoreReadErrorWithClusterBreakingPotentiallabelSelector (in query): stringlabelSelectorlimit (in query): integerlimitpretty (in query): stringprettypropagationPolicy (in query): stringpropagationPolicyresourceVersion (in query): stringresourceVersionresourceVersionMatch (in query): stringresourceVersionMatchsendInitialEvents (in query): booleansendInitialEventstimeoutSeconds (in query): integertimeoutSecondsResponse200 (Status): OK401: UnauthorizedThis page is automatically generated.If you plan to report an issue with this page, mention that the page is auto-generated in your issue description. The fix may need to happen elsewhere in the Kubernetes project.


# Participating in SIG Docs

Participating in SIG DocsSIG Docs is one of the special interest groups within the Kubernetes project, focused on writing, updating, and maintaining the documentation for Kubernetes as a whole. See SIG Docs from the community github repo for more information about the SIG.SIG Docs welcomes content and reviews from all contributors. Anyone can open a pull request (PR), and anyone is welcome to file issues about content or comment on pull requests in progress.You can also become a member, reviewer, or approver. These roles require greater access and entail certain responsibilities for approving and committing changes. See community-membership for more information on how membership works within the Kubernetes community.The rest of this document outlines some unique ways these roles function within SIG Docs, which is responsible for maintaining one of the most public-facing aspects of Kubernetes -- the Kubernetes website and documentation.SIG Docs chairpersonEach SIG, including SIG Docs, selects one or more SIG members to act as chairpersons. These are points of contact between SIG Docs and other parts of the Kubernetes organization. They require extensive knowledge of the structure of the Kubernetes project as a whole and how SIG Docs works within it. See Leadership for the current list of chairpersons.SIG Docs teams and automationAutomation in SIG Docs relies on two different mechanisms: GitHub teams and OWNERS files.GitHub teamsThere are two categories of SIG Docs teams on GitHub:@sig-docs-{language}-owners are approvers and leads@sig-docs-{language}-reviews are reviewersEach can be referenced with their @name in GitHub comments to communicate with everyone in that group.Sometimes Prow and GitHub teams overlap without matching exactly. For assignment of issues, pull requests, and to support PR approvals, the automation uses information from OWNERS files.OWNERS files and front-matterThe Kubernetes project uses an automation tool called prow for automation related to GitHub issues and pull requests. The Kubernetes website repository uses two prow plugins:blunderbussapproveThese two plugins use the OWNERS and OWNERS_ALIASES files in the top level of the kubernetes/website GitHub repository to control how prow works within the repository.An OWNERS file contains a list of people who are SIG Docs reviewers and approvers. OWNERS files can also exist in subdirectories, and can override who can act as a reviewer or approver of files in that subdirectory and its descendants. For more information about OWNERS files in general, see OWNERS.In addition, an individual Markdown file can list reviewers and approvers in its front-matter, either by listing individual GitHub usernames or GitHub groups.The combination of OWNERS files and front-matter in Markdown files determines the advice PR owners get from automated systems about who to ask for technical and editorial review of their PR.How merging worksWhen a pull request is merged to the branch used to publish content, that content is published to https://kubernetes.io. To ensure that the quality of our published content is high, we limit merging pull requests to SIG Docs approvers. Here's how it works.When a pull request has both the lgtm and approve labels, has no hold labels, and all tests are passing, the pull request merges automatically.Kubernetes organization members and SIG Docs approvers can add comments to prevent automatic merging of a given pull request (by adding a /hold comment or withholding a /lgtm comment).Any Kubernetes member can add the lgtm label by adding a /lgtm comment.Only SIG Docs approvers can merge a pull request by adding an /approve comment. Some approvers also perform additional specific roles, such as PR Wrangler or SIG Docs chairperson.What's nextFor more information about contributing to the Kubernetes documentation, see:Contributing new contentReviewing contentDocumentation style guideRoles and responsibilitiesIssue WranglersPR wranglers


# Pod Priority and Preemption

Pod Priority and PreemptionFEATURE STATE: Kubernetes v1.14 [stable]Pods can have priority. Priority indicates the importance of a Pod relative to other Pods. If a Pod cannot be scheduled, the scheduler tries to preempt (evict) lower priority Pods to make scheduling of the pending Pod possible.Warning:In a cluster where not all users are trusted, a malicious user could create Pods at the highest possible priorities, causing other Pods to be evicted/not get scheduled. An administrator can use ResourceQuota to prevent users from creating pods at high priorities.See limit Priority Class consumption by default for details.How to use priority and preemptionTo use priority and preemption:Add one or more PriorityClasses.Create Pods withpriorityClassName set to one of the added PriorityClasses. Of course you do not need to create the Pods directly; normally you would add priorityClassName to the Pod template of a collection object like a Deployment.Keep reading for more information about these steps.Note:Kubernetes already ships with two PriorityClasses: system-cluster-critical and system-node-critical. These are common classes and are used to ensure that critical components are always scheduled first.PriorityClassA PriorityClass is a non-namespaced object that defines a mapping from a priority class name to the integer value of the priority. The name is specified in the name field of the PriorityClass object's metadata. The value is specified in the required value field. The higher the value, the higher the priority. The name of a PriorityClass object must be a valid DNS subdomain name, and it cannot be prefixed with system-.A PriorityClass object can have any 32-bit integer value smaller than or equal to 1 billion. This means that the range of values for a PriorityClass object is from -2147483648 to 1000000000 inclusive. Larger numbers are reserved for built-in PriorityClasses that represent critical system Pods. A cluster admin should create one PriorityClass object for each such mapping that they want.PriorityClass also has two optional fields: globalDefault and description. The globalDefault field indicates that the value of this PriorityClass should be used for Pods without a priorityClassName. Only one PriorityClass with globalDefault set to true can exist in the system. If there is no PriorityClass with globalDefault set, the priority of Pods with no priorityClassName is zero.The description field is an arbitrary string. It is meant to tell users of the cluster when they should use this PriorityClass.Notes about PodPriority and existing clustersIf you upgrade an existing cluster without this feature, the priority of your existing Pods is effectively zero.Addition of a PriorityClass with globalDefault set to true does not change the priorities of existing Pods. The value of such a PriorityClass is used only for Pods created after the PriorityClass is added.If you delete a PriorityClass, existing Pods that use the name of the deleted PriorityClass remain unchanged, but you cannot create more Pods that use the name of the deleted PriorityClass.Example PriorityClassapiVersion: scheduling.k8s.io/v1 kind: PriorityClass metadata: name: high-priority value: 1000000 globalDefault: false description: "This priority class should be used for XYZ service pods only." Non-preempting PriorityClassFEATURE STATE: Kubernetes v1.24 [stable]Pods with preemptionPolicy: Never will be placed in the scheduling queue ahead of lower-priority pods, but they cannot preempt other pods. A non-preempting pod waiting to be scheduled will stay in the scheduling queue, until sufficient resources are free, and it can be scheduled. Non-preempting pods, like other pods, are subject to scheduler back-off. This means that if the scheduler tries these pods and they cannot be scheduled, they will be retried with lower frequency, allowing other pods with lower priority to be scheduled before them.Non-preempting pods may still be preempted by other, high-priority pods.preemptionPolicy defaults to PreemptLowerPriority, which will allow pods of that PriorityClass to preempt lower-priority pods (as is existing default behavior). If preemptionPolicy is set to Never, pods in that PriorityClass will be non-preempting.An example use case is for data science workloads. A user may submit a job that they want to be prioritized above other workloads, but do not wish to discard existing work by preempting running pods. The high priority job with preemptionPolicy: Never will be scheduled ahead of other queued pods, as soon as sufficient cluster resources "naturally" become free.Example Non-preempting PriorityClassapiVersion: scheduling.k8s.io/v1 kind: PriorityClass metadata: name: high-priority-nonpreempting value: 1000000 preemptionPolicy: Never globalDefault: false description: "This priority class will not cause other pods to be preempted." Pod priorityAfter you have one or more PriorityClasses, you can create Pods that specify one of those PriorityClass names in their specifications. The priority admission controller uses the priorityClassName field and populates the integer value of the priority. If the priority class is not found, the Pod is rejected.The following YAML is an example of a Pod configuration that uses the PriorityClass created in the preceding example. The priority admission controller checks the specification and resolves the priority of the Pod to 1000000.apiVersion: v1 kind: Pod metadata: name: nginx labels: env: test spec: containers: - name: nginx image: nginx imagePullPolicy: IfNotPresent priorityClassName: high-priority Effect of Pod priority on scheduling orderWhen Pod priority is enabled, the scheduler orders pending Pods by their priority and a pending Pod is placed ahead of other pending Pods with lower priority in the scheduling queue. As a result, the higher priority Pod may be scheduled sooner than Pods with lower priority if its scheduling requirements are met. If such Pod cannot be scheduled, the scheduler will continue and try to schedule other lower priority Pods.PreemptionWhen Pods are created, they go to a queue and wait to be scheduled. The scheduler picks a Pod from the queue and tries to schedule it on a Node. If no Node is found that satisfies all the specified requirements of the Pod, preemption logic is triggered for the pending Pod. Let's call the pending Pod P. Preemption logic tries to find a Node where removal of one or more Pods with lower priority than P would enable P to be scheduled on that Node. If such a Node is found, one or more lower priority Pods get evicted from the Node. After the Pods are gone, P can be scheduled on the Node.User exposed informationWhen Pod P preempts one or more Pods on Node N, nominatedNodeName field of Pod P's status is set to the name of Node N. This field helps the scheduler track resources reserved for Pod P and also gives users information about preemptions in their clusters.Please note that Pod P is not necessarily scheduled to the "nominated Node". The scheduler always tries the "nominated Node" before iterating over any other nodes. After victim Pods are preempted, they get their graceful termination period. If another node becomes available while scheduler is waiting for the victim Pods to terminate, scheduler may use the other node to schedule Pod P. As a result nominatedNodeName and nodeName of Pod spec are not always the same. Also, if the scheduler preempts Pods on Node N, but then a higher priority Pod than Pod P arrives, the scheduler may give Node N to the new higher priority Pod. In such a case, scheduler clears nominatedNodeName of Pod P. By doing this, scheduler makes Pod P eligible to preempt Pods on another Node.Limitations of preemptionGraceful termination of preemption victimsWhen Pods are preempted, the victims get their graceful termination period. They have that much time to finish their work and exit. If they don't, they are killed. This graceful termination period creates a time gap between the point that the scheduler preempts Pods and the time when the pending Pod (P) can be scheduled on the Node (N). In the meantime, the scheduler keeps scheduling other pending Pods. As victims exit or get terminated, the scheduler tries to schedule Pods in the pending queue. Therefore, there is usually a time gap between the point that scheduler preempts victims and the time that Pod P is scheduled. In order to minimize this gap, one can set graceful termination period of lower priority Pods to zero or a small number.PodDisruptionBudget is supported, but not guaranteedA PodDisruptionBudget (PDB) allows application owners to limit the number of Pods of a replicated application that are down simultaneously from voluntary disruptions. Kubernetes supports PDB when preempting Pods, but respecting PDB is best effort. The scheduler tries to find victims whose PDB are not violated by preemption, but if no such victims are found, preemption will still happen, and lower priority Pods will be removed despite their PDBs being violated.Inter-Pod affinity on lower-priority PodsA Node is considered for preemption only when the answer to this question is yes: "If all the Pods with lower priority than the pending Pod are removed from the Node, can the pending Pod be scheduled on the Node?"Note:Preemption does not necessarily remove all lower-priority Pods. If the pending Pod can be scheduled by removing fewer than all lower-priority Pods, then only a portion of the lower-priority Pods are removed. Even so, the answer to the preceding question must be yes. If the answer is no, the Node is not considered for preemption.If a pending Pod has inter-pod affinity to one or more of the lower-priority Pods on the Node, the inter-Pod affinity rule cannot be satisfied in the absence of those lower-priority Pods. In this case, the scheduler does not preempt any Pods on the Node. Instead, it looks for another Node. The scheduler might find a suitable Node or it might not. There is no guarantee that the pending Pod can be scheduled.Our recommended solution for this problem is to create inter-Pod affinity only towards equal or higher priority Pods.Cross node preemptionSuppose a Node N is being considered for preemption so that a pending Pod P can be scheduled on N. P might become feasible on N only if a Pod on another Node is preempted. Here's an example:Pod P is being considered for Node N.Pod Q is running on another Node in the same Zone as Node N.Pod P has Zone-wide anti-affinity with Pod Q (topologyKey: topology.kubernetes.io/zone).There are no other cases of anti-affinity between Pod P and other Pods in the Zone.In order to schedule Pod P on Node N, Pod Q can be preempted, but scheduler does not perform cross-node preemption. So, Pod P will be deemed unschedulable on Node N.If Pod Q were removed from its Node, the Pod anti-affinity violation would be gone, and Pod P could possibly be scheduled on Node N.We may consider adding cross Node preemption in future versions if there is enough demand and if we find an algorithm with reasonable performance.TroubleshootingPod priority and preemption can have unwanted side effects. Here are some examples of potential problems and ways to deal with them.Pods are preempted unnecessarilyPreemption removes existing Pods from a cluster under resource pressure to make room for higher priority pending Pods. If you give high priorities to certain Pods by mistake, these unintentionally high priority Pods may cause preemption in your cluster. Pod priority is specified by setting the priorityClassName field in the Pod's specification. The integer value for priority is then resolved and populated to the priority field of podSpec.To address the problem, you can change the priorityClassName for those Pods to use lower priority classes, or leave that field empty. An empty priorityClassName is resolved to zero by default.When a Pod is preempted, there will be events recorded for the preempted Pod. Preemption should happen only when a cluster does not have enough resources for a Pod. In such cases, preemption happens only when the priority of the pending Pod (preemptor) is higher than the victim Pods. Preemption must not happen when there is no pending Pod, or when the pending Pods have equal or lower priority than the victims. If preemption happens in such scenarios, please file an issue.Pods are preempted, but the preemptor is not scheduledWhen pods are preempted, they receive their requested graceful termination period, which is by default 30 seconds. If the victim Pods do not terminate within this period, they are forcibly terminated. Once all the victims go away, the preemptor Pod can be scheduled.While the preemptor Pod is waiting for the victims to go away, a higher priority Pod may be created that fits on the same Node. In this case, the scheduler will schedule the higher priority Pod instead of the preemptor.This is expected behavior: the Pod with the higher priority should take the place of a Pod with a lower priority.Higher priority Pods are preempted before lower priority podsThe scheduler tries to find nodes that can run a pending Pod. If no node is found, the scheduler tries to remove Pods with lower priority from an arbitrary node in order to make room for the pending pod. If a node with low priority Pods is not feasible to run the pending Pod, the scheduler may choose another node with higher priority Pods (compared to the Pods on the other node) for preemption. The victims must still have lower priority than the preemptor Pod.When there are multiple nodes available for preemption, the scheduler tries to choose the node with a set of Pods with lowest priority. However, if such Pods have PodDisruptionBudget that would be violated if they are preempted then the scheduler may choose another node with higher priority Pods.When multiple nodes exist for preemption and none of the above scenarios apply, the scheduler chooses a node with the lowest priority.Interactions between Pod priority and quality of servicePod priority and QoS class are two orthogonal features with few interactions and no default restrictions on setting the priority of a Pod based on its QoS classes. The scheduler's preemption logic does not consider QoS when choosing preemption targets. Preemption considers Pod priority and attempts to choose a set of targets with the lowest priority. Higher-priority Pods are considered for preemption only if the removal of the lowest priority Pods is not sufficient to allow the scheduler to schedule the preemptor Pod, or if the lowest priority Pods are protected by PodDisruptionBudget.The kubelet uses Priority to determine pod order for node-pressure eviction. You can use the QoS class to estimate the order in which pods are most likely to get evicted. The kubelet ranks pods for eviction based on the following factors:Whether the starved resource usage exceeds requestsPod PriorityAmount of resource usage relative to requestsSee Pod selection for kubelet eviction for more details.kubelet node-pressure eviction does not evict Pods when their usage does not exceed their requests. If a Pod with lower priority is not exceeding its requests, it won't be evicted. Another Pod with higher priority that exceeds its requests may be evicted.What's nextRead about using ResourceQuotas in connection with PriorityClasses: limit Priority Class consumption by defaultLearn about Pod DisruptionLearn about API-initiated EvictionLearn about Node-pressure Eviction


# Event Rate Limit Configuration (v1alpha1)

Event Rate Limit Configuration (v1alpha1)Resource TypesConfigurationConfigurationConfiguration provides configuration for the EventRateLimit admission controller.FieldDescriptionapiVersionstringeventratelimit.admission.k8s.io/v1alpha1kindstringConfigurationlimits [Required][]Limitlimits are the limits to place on event queries received. Limits can be placed on events received server-wide, per namespace, per user, and per source+object. At least one limit is required.LimitAppears in:ConfigurationLimit is the configuration for a particular limit typeFieldDescriptiontype [Required]LimitTypetype is the type of limit to which this configuration appliesqps [Required]int32qps is the number of event queries per second that are allowed for this type of limit. The qps and burst fields are used together to determine if a particular event query is accepted. The qps determines how many queries are accepted once the burst amount of queries has been exhausted.burst [Required]int32burst is the burst number of event queries that are allowed for this type of limit. The qps and burst fields are used together to determine if a particular event query is accepted. The burst determines the maximum size of the allowance granted for a particular bucket. For example, if the burst is 10 and the qps is 3, then the admission control will accept 10 queries before blocking any queries. Every second, 3 more queries will be allowed. If some of that allowance is not used, then it will roll over to the next second, until the maximum allowance of 10 is reached.cacheSizeint32cacheSize is the size of the LRU cache for this type of limit. If a bucket is evicted from the cache, then the allowance for that bucket is reset. If more queries are later received for an evicted bucket, then that bucket will re-enter the cache with a clean slate, giving that bucket a full allowance of burst queries.The default cache size is 4096.If limitType is 'server', then cacheSize is ignored.LimitType(Alias of string)Appears in:LimitLimitType is the type of the limit (e.g., per-namespace)This page is automatically generated.If you plan to report an issue with this page, mention that the page is auto-generated in your issue description. The fix may need to happen elsewhere in the Kubernetes project.


# Allocate Devices to Workloads with DRA

Allocate Devices to Workloads with DRAFEATURE STATE: Kubernetes v1.34 [stable] (enabled by default: true)This page shows you how to allocate devices to your Pods by using dynamic resource allocation (DRA). These instructions are for workload operators. Before reading this page, familiarize yourself with how DRA works and with DRA terminology like ResourceClaims and ResourceClaimTemplates. For more information, see Dynamic Resource Allocation (DRA).About device allocation with DRAAs a workload operator, you can claim devices for your workloads by creating ResourceClaims or ResourceClaimTemplates. When you deploy your workload, Kubernetes and the device drivers find available devices, allocate them to your Pods, and place the Pods on nodes that can access those devices.Before you beginYou need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. It is recommended to run this tutorial on a cluster with at least two nodes that are not acting as control plane hosts. If you do not already have a cluster, you can create one by using minikube or you can use one of these Kubernetes playgrounds:iximiuz LabsKillercodaKodeKloudPlay with KubernetesYour Kubernetes server must be version v1.34.To check the version, enter kubectl version.Ensure that your cluster admin has set up DRA, attached devices, and installed drivers. For more information, see Set Up DRA in a Cluster.Identify devices to claimYour cluster administrator or the device drivers create DeviceClasses that define categories of devices. You can claim devices by using Common Expression Language to filter for specific device properties.Get a list of DeviceClasses in the cluster:kubectl get deviceclasses The output is similar to the following:NAME AGE driver.example.com 16m If you get a permission error, you might not have access to get DeviceClasses. Check with your cluster administrator or with the driver provider for available device properties.Claim resourcesYou can request resources from a DeviceClass by using ResourceClaims. To create a ResourceClaim, do one of the following:Manually create a ResourceClaim if you want multiple Pods to share access to the same devices, or if you want a claim to exist beyond the lifetime of a Pod.Use a ResourceClaimTemplate to let Kubernetes generate and manage per-Pod ResourceClaims. Create a ResourceClaimTemplate if you want every Pod to have access to separate devices that have similar configurations. For example, you might want simultaneous access to devices for Pods in a Job that uses parallel execution.If you directly reference a specific ResourceClaim in a Pod, that ResourceClaim must already exist in the cluster. If a referenced ResourceClaim doesn't exist, the Pod remains in a pending state until the ResourceClaim is created. You can reference an auto-generated ResourceClaim in a Pod, but this isn't recommended because auto-generated ResourceClaims are bound to the lifetime of the Pod that triggered the generation.To create a workload that claims resources, select one of the following options:Review the following example manifest:dra/resourceclaimtemplate.yaml apiVersion: resource.k8s.io/v1 kind: ResourceClaimTemplate metadata: name: example-resource-claim-template spec: spec: devices: requests: - name: gpu-claim exactly: deviceClassName: example-device-class selectors: - cel: expression: |- device.attributes["driver.example.com"].type == "gpu" && device.capacity["driver.example.com"].memory == quantity("64Gi") This manifest creates a ResourceClaimTemplate that requests devices in the example-device-class DeviceClass that match both of the following parameters:Devices that have a driver.example.com/type attribute with a value of gpu.Devices that have 64Gi of capacity.To create the ResourceClaimTemplate, run the following command:kubectl apply -f https://k8s.io/examples/dra/resourceclaimtemplate.yaml Review the following example manifest:dra/resourceclaim.yaml apiVersion: resource.k8s.io/v1 kind: ResourceClaim metadata: name: example-resource-claim spec: devices: requests: - name: single-gpu-claim exactly: deviceClassName: example-device-class allocationMode: All selectors: - cel: expression: |- device.attributes["driver.example.com"].type == "gpu" && device.capacity["driver.example.com"].memory == quantity("64Gi") This manifest creates ResourceClaim that requests devices in the example-device-class DeviceClass that match both of the following parameters:Devices that have a driver.example.com/type attribute with a value of gpu.Devices that have 64Gi of capacity.To create the ResourceClaim, run the following command:kubectl apply -f https://k8s.io/examples/dra/resourceclaim.yaml Request devices in workloads using DRATo request device allocation, specify a ResourceClaim or a ResourceClaimTemplate in the resourceClaims field of the Pod specification. Then, request a specific claim by name in the resources.claims field of a container in that Pod. You can specify multiple entries in the resourceClaims field and use specific claims in different containers.Review the following example Job:dra/dra-example-job.yaml apiVersion: batch/v1 kind: Job metadata: name: example-dra-job spec: completions: 10 parallelism: 2 template: spec: restartPolicy: Never containers: - name: container0 image: ubuntu:24.04 command: ["sleep", "9999"] resources: claims: - name: separate-gpu-claim - name: container1 image: ubuntu:24.04 command: ["sleep", "9999"] resources: claims: - name: shared-gpu-claim - name: container2 image: ubuntu:24.04 command: ["sleep", "9999"] resources: claims: - name: shared-gpu-claim resourceClaims: - name: separate-gpu-claim resourceClaimTemplateName: example-resource-claim-template - name: shared-gpu-claim resourceClaimName: example-resource-claim Each Pod in this Job has the following properties:Makes a ResourceClaimTemplate named separate-gpu-claim and a ResourceClaim named shared-gpu-claim available to containers.Runs the following containers:container0 requests the devices from the separate-gpu-claim ResourceClaimTemplate.container1 and container2 share access to the devices from the shared-gpu-claim ResourceClaim.Create the Job:kubectl apply -f https://k8s.io/examples/dra/dra-example-job.yaml Try the following troubleshooting steps:When the workload does not start as expected, drill down from Job to Pods to ResourceClaims and check the objects at each level with kubectl describe to see whether there are any status fields or events which might explain why the workload is not starting.When creating a Pod fails with must specify one of: resourceClaimName, resourceClaimTemplateName, check that all entries in pod.spec.resourceClaims have exactly one of those fields set. If they do, then it is possible that the cluster has a mutating Pod webhook installed which was built against APIs from Kubernetes < 1.32. Work with your cluster administrator to check this.Clean upTo delete the Kubernetes objects that you created in this task, follow these steps:Delete the example Job:kubectl delete -f https://k8s.io/examples/dra/dra-example-job.yaml To delete your resource claims, run one of the following commands:Delete the ResourceClaimTemplate:kubectl delete -f https://k8s.io/examples/dra/resourceclaimtemplate.yaml Delete the ResourceClaim:kubectl delete -f https://k8s.io/examples/dra/resourceclaim.yaml What's nextLearn more about DRA


# Using Source IP

Using Source IPApplications running in a Kubernetes cluster find and communicate with each other, and the outside world, through the Service abstraction. This document explains what happens to the source IP of packets sent to different types of Services, and how you can toggle this behavior according to your needs.Before you beginTerminologyThis document makes use of the following terms:NATNetwork address translationSource NATReplacing the source IP on a packet; in this page, that usually means replacing with the IP address of a node.Destination NATReplacing the destination IP on a packet; in this page, that usually means replacing with the IP address of a PodVIPA virtual IP address, such as the one assigned to every Service in Kuberneteskube-proxyA network daemon that orchestrates Service VIP management on every nodePrerequisitesYou need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. It is recommended to run this tutorial on a cluster with at least two nodes that are not acting as control plane hosts. If you do not already have a cluster, you can create one by using minikube or you can use one of these Kubernetes playgrounds:iximiuz LabsKillercodaKodeKloudPlay with KubernetesThe examples use a small nginx webserver that echoes back the source IP of requests it receives through an HTTP header. You can create it as follows:Note:The image in the following command only runs on AMD64 architectures.kubectl create deployment source-ip-app --image=registry.k8s.io/echoserver:1.10 The output is:deployment.apps/source-ip-app created ObjectivesExpose a simple application through various types of ServicesUnderstand how each Service type handles source IP NATUnderstand the tradeoffs involved in preserving source IPSource IP for Services with Type=ClusterIPPackets sent to ClusterIP from within the cluster are never source NAT'd if you're running kube-proxy in iptables mode, (the default). You can query the kube-proxy mode by fetching http://localhost:10249/proxyMode on the node where kube-proxy is running.kubectl get nodes The output is similar to this:NAME STATUS ROLES AGE VERSION kubernetes-node-6jst Ready <none> 2h v1.13.0 kubernetes-node-cx31 Ready <none> 2h v1.13.0 kubernetes-node-jj1t Ready <none> 2h v1.13.0 Get the proxy mode on one of the nodes (kube-proxy listens on port 10249):# Run this in a shell on the node you want to query. curl http://localhost:10249/proxyMode The output is:iptables You can test source IP preservation by creating a Service over the source IP app:kubectl expose deployment source-ip-app --name=clusterip --port=80 --target-port=8080 The output is:service/clusterip exposed kubectl get svc clusterip The output is similar to:NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE clusterip ClusterIP 10.0.170.92 <none> 80/TCP 51s And hitting the ClusterIP from a pod in the same cluster:kubectl run busybox -it --image=busybox:1.28 --restart=Never --rm The output is similar to this:Waiting for pod default/busybox to be running, status is Pending, pod ready: false If you don't see a command prompt, try pressing enter. You can then run a command inside that Pod:# Run this inside the terminal from "kubectl run" ip addr 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 3: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1460 qdisc noqueue link/ether 0a:58:0a:f4:03:08 brd ff:ff:ff:ff:ff:ff inet 10.244.3.8/24 scope global eth0 valid_lft forever preferred_lft forever inet6 fe80::188a:84ff:feb0:26a5/64 scope link valid_lft forever preferred_lft forever …then use wget to query the local webserver# Replace "10.0.170.92" with the IPv4 address of the Service named "clusterip" wget -qO - 10.0.170.92 CLIENT VALUES: client_address=10.244.3.8 command=GET ... The client_address is always the client pod's IP address, whether the client pod and server pod are in the same node or in different nodes.Source IP for Services with Type=NodePortPackets sent to Services with Type=NodePort are source NAT'd by default. You can test this by creating a NodePort Service:kubectl expose deployment source-ip-app --name=nodeport --port=80 --target-port=8080 --type=NodePort The output is:service/nodeport exposed NODEPORT=$(kubectl get -o jsonpath="{.spec.ports[0].nodePort}" services nodeport) NODES=$(kubectl get nodes -o jsonpath='{ $.items[*].status.addresses[?(@.type=="InternalIP")].address }') If you're running on a cloud provider, you may need to open up a firewall-rule for the nodes:nodeport reported above. Now you can try reaching the Service from outside the cluster through the node port allocated above.for node in $NODES; do curl -s $node:$NODEPORT | grep -i client_address; done The output is similar to:client_address=10.180.1.1 client_address=10.240.0.5 client_address=10.240.0.3 Note that these are not the correct client IPs, they're cluster internal IPs. This is what happens:Client sends packet to node2:nodePortnode2 replaces the source IP address (SNAT) in the packet with its own IP addressnode2 replaces the destination IP on the packet with the pod IPpacket is routed to node 1, and then to the endpointthe pod's reply is routed back to node2the pod's reply is sent back to the clientVisually:Figure. Source IP Type=NodePort using SNATTo avoid this, Kubernetes has a feature to preserve the client source IP. If you set service.spec.externalTrafficPolicy to the value Local, kube-proxy only proxies proxy requests to local endpoints, and does not forward traffic to other nodes. This approach preserves the original source IP address. If there are no local endpoints, packets sent to the node are dropped, so you can rely on the correct source-ip in any packet processing rules you might apply a packet that make it through to the endpoint.Set the service.spec.externalTrafficPolicy field as follows:kubectl patch svc nodeport -p '{"spec":{"externalTrafficPolicy":"Local"}}' The output is:service/nodeport patched Now, re-run the test:for node in $NODES; do curl --connect-timeout 1 -s $node:$NODEPORT | grep -i client_address; done The output is similar to:client_address=198.51.100.79 Note that you only got one reply, with the right client IP, from the one node on which the endpoint pod is running.This is what happens:client sends packet to node2:nodePort, which doesn't have any endpointspacket is droppedclient sends packet to node1:nodePort, which does have endpointsnode1 routes packet to endpoint with the correct source IPVisually:Figure. Source IP Type=NodePort preserves client source IP addressSource IP for Services with Type=LoadBalancerPackets sent to Services with Type=LoadBalancer are source NAT'd by default, because all schedulable Kubernetes nodes in the Ready state are eligible for load-balanced traffic. So if packets arrive at a node without an endpoint, the system proxies it to a node with an endpoint, replacing the source IP on the packet with the IP of the node (as described in the previous section).You can test this by exposing the source-ip-app through a load balancer:kubectl expose deployment source-ip-app --name=loadbalancer --port=80 --target-port=8080 --type=LoadBalancer The output is:service/loadbalancer exposed Print out the IP addresses of the Service:kubectl get svc loadbalancer The output is similar to this:NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE loadbalancer LoadBalancer 10.0.65.118 203.0.113.140 80/TCP 5m Next, send a request to this Service's external-ip:curl 203.0.113.140 The output is similar to this:CLIENT VALUES: client_address=10.240.0.5 ... However, if you're running on Google Kubernetes Engine/GCE, setting the same service.spec.externalTrafficPolicy field to Local forces nodes without Service endpoints to remove themselves from the list of nodes eligible for loadbalanced traffic by deliberately failing health checks.Visually:You can test this by setting the annotation:kubectl patch svc loadbalancer -p '{"spec":{"externalTrafficPolicy":"Local"}}' You should immediately see the service.spec.healthCheckNodePort field allocated by Kubernetes:kubectl get svc loadbalancer -o yaml | grep -i healthCheckNodePort The output is similar to this: healthCheckNodePort: 32122 The service.spec.healthCheckNodePort field points to a port on every node serving the health check at /healthz. You can test this:kubectl get pod -o wide -l app=source-ip-app The output is similar to this:NAME READY STATUS RESTARTS AGE IP NODE source-ip-app-826191075-qehz4 1/1 Running 0 20h 10.180.1.136 kubernetes-node-6jst Use curl to fetch the /healthz endpoint on various nodes:# Run this locally on a node you choose curl localhost:32122/healthz 1 Service Endpoints found On a different node you might get a different result:# Run this locally on a node you choose curl localhost:32122/healthz No Service Endpoints Found A controller running on the control plane is responsible for allocating the cloud load balancer. The same controller also allocates HTTP health checks pointing to this port/path on each node. Wait about 10 seconds for the 2 nodes without endpoints to fail health checks, then use curl to query the IPv4 address of the load balancer:curl 203.0.113.140 The output is similar to this:CLIENT VALUES: client_address=198.51.100.79 ... Cross-platform supportOnly some cloud providers offer support for source IP preservation through Services with Type=LoadBalancer. The cloud provider you're running on might fulfill the request for a loadbalancer in a few different ways:With a proxy that terminates the client connection and opens a new connection to your nodes/endpoints. In such cases the source IP will always be that of the cloud LB, not that of the client.With a packet forwarder, such that requests from the client sent to the loadbalancer VIP end up at the node with the source IP of the client, not an intermediate proxy.Load balancers in the first category must use an agreed upon protocol between the loadbalancer and backend to communicate the true client IP such as the HTTP Forwarded or X-FORWARDED-FOR headers, or the proxy protocol. Load balancers in the second category can leverage the feature described above by creating an HTTP health check pointing at the port stored in the service.spec.healthCheckNodePort field on the Service.Cleaning upDelete the Services:kubectl delete svc -l app=source-ip-app Delete the Deployment, ReplicaSet and Pod:kubectl delete deployment source-ip-app What's nextLearn more about connecting applications via servicesRead how to Create an External Load Balancer


# Updating Reference Documentation

Updating Reference DocumentationThe topics in this section document how to generate the Kubernetes reference guides.To build the reference documentation, see the following guide:Generating Reference Documentation QuickstartContributing to the Upstream Kubernetes CodeGenerating Reference Documentation for the Kubernetes APIGenerating Reference Documentation for kubectl CommandsGenerating Reference Documentation for MetricsGenerating Reference Pages for Kubernetes Components and Tools


# Service Accounts

Service AccountsLearn about ServiceAccount objects in Kubernetes.This page introduces the ServiceAccount object in Kubernetes, providing information about how service accounts work, use cases, limitations, alternatives, and links to resources for additional guidance.What are service accounts?A service account is a type of non-human account that, in Kubernetes, provides a distinct identity in a Kubernetes cluster. Application Pods, system components, and entities inside and outside the cluster can use a specific ServiceAccount's credentials to identify as that ServiceAccount. This identity is useful in various situations, including authenticating to the API server or implementing identity-based security policies.Service accounts exist as ServiceAccount objects in the API server. Service accounts have the following properties:Namespaced: Each service account is bound to a Kubernetes namespace. Every namespace gets a default ServiceAccount upon creation.Lightweight: Service accounts exist in the cluster and are defined in the Kubernetes API. You can quickly create service accounts to enable specific tasks.Portable: A configuration bundle for a complex containerized workload might include service account definitions for the system's components. The lightweight nature of service accounts and the namespaced identities make the configurations portable.Service accounts are different from user accounts, which are authenticated human users in the cluster. By default, user accounts don't exist in the Kubernetes API server; instead, the API server treats user identities as opaque data. You can authenticate as a user account using multiple methods. Some Kubernetes distributions might add custom extension APIs to represent user accounts in the API server.Comparison between service accounts and usersDescriptionServiceAccountUser or groupLocationKubernetes API (ServiceAccount object)ExternalAccess controlKubernetes RBAC or other authorization mechanismsKubernetes RBAC or other identity and access management mechanismsIntended useWorkloads, automationPeopleDefault service accountsWhen you create a cluster, Kubernetes automatically creates a ServiceAccount object named default for every namespace in your cluster. The default service accounts in each namespace get no permissions by default other than the default API discovery permissions that Kubernetes grants to all authenticated principals if role-based access control (RBAC) is enabled. If you delete the default ServiceAccount object in a namespace, the control plane replaces it with a new one.If you deploy a Pod in a namespace, and you don't manually assign a ServiceAccount to the Pod, Kubernetes assigns the default ServiceAccount for that namespace to the Pod.Use cases for Kubernetes service accountsAs a general guideline, you can use service accounts to provide identities in the following scenarios:Your Pods need to communicate with the Kubernetes API server, for example in situations such as the following:Providing read-only access to sensitive information stored in Secrets.Granting cross-namespace access, such as allowing a Pod in namespace example to read, list, and watch for Lease objects in the kube-node-lease namespace.Your Pods need to communicate with an external service. For example, a workload Pod requires an identity for a commercially available cloud API, and the commercial provider allows configuring a suitable trust relationship.Authenticating to a private image registry using an imagePullSecret.An external service needs to communicate with the Kubernetes API server. For example, authenticating to the cluster as part of a CI/CD pipeline.You use third-party security software in your cluster that relies on the ServiceAccount identity of different Pods to group those Pods into different contexts.How to use service accountsTo use a Kubernetes service account, you do the following:Create a ServiceAccount object using a Kubernetes client like kubectl or a manifest that defines the object.Grant permissions to the ServiceAccount object using an authorization mechanism such as RBAC.Assign the ServiceAccount object to Pods during Pod creation.If you're using the identity from an external service, retrieve the ServiceAccount token and use it from that service instead.For instructions, refer to Configure Service Accounts for Pods.Grant permissions to a ServiceAccountYou can use the built-in Kubernetes role-based access control (RBAC) mechanism to grant the minimum permissions required by each service account. You create a role, which grants access, and then bind the role to your ServiceAccount. RBAC lets you define a minimum set of permissions so that the service account permissions follow the principle of least privilege. Pods that use that service account don't get more permissions than are required to function correctly.For instructions, refer to ServiceAccount permissions.Cross-namespace access using a ServiceAccountYou can use RBAC to allow service accounts in one namespace to perform actions on resources in a different namespace in the cluster. For example, consider a scenario where you have a service account and Pod in the dev namespace and you want your Pod to see Jobs running in the maintenance namespace. You could create a Role object that grants permissions to list Job objects. Then, you'd create a RoleBinding object in the maintenance namespace to bind the Role to the ServiceAccount object. Now, Pods in the dev namespace can list Job objects in the maintenance namespace using that service account.Assign a ServiceAccount to a PodTo assign a ServiceAccount to a Pod, you set the spec.serviceAccountName field in the Pod specification. Kubernetes then automatically provides the credentials for that ServiceAccount to the Pod. In v1.22 and later, Kubernetes gets a short-lived, automatically rotating token using the TokenRequest API and mounts the token as a projected volume.By default, Kubernetes provides the Pod with the credentials for an assigned ServiceAccount, whether that is the default ServiceAccount or a custom ServiceAccount that you specify.To prevent Kubernetes from automatically injecting credentials for a specified ServiceAccount or the default ServiceAccount, set the automountServiceAccountToken field in your Pod specification to false.In versions earlier than 1.22, Kubernetes provides a long-lived, static token to the Pod as a Secret.Manually retrieve ServiceAccount credentialsIf you need the credentials for a ServiceAccount to mount in a non-standard location, or for an audience that isn't the API server, use one of the following methods:TokenRequest API (recommended): Request a short-lived service account token from within your own application code. The token expires automatically and can rotate upon expiration. If you have a legacy application that is not aware of Kubernetes, you could use a sidecar container within the same pod to fetch these tokens and make them available to the application workload.Token Volume Projection (also recommended): In Kubernetes v1.20 and later, use the Pod specification to tell the kubelet to add the service account token to the Pod as a projected volume. Projected tokens expire automatically, and the kubelet rotates the token before it expires.Service Account Token Secrets (not recommended): You can mount service account tokens as Kubernetes Secrets in Pods. These tokens don't expire and don't rotate. In versions prior to v1.24, a permanent token was automatically created for each service account. This method is not recommended anymore, especially at scale, because of the risks associated with static, long-lived credentials. The LegacyServiceAccountTokenNoAutoGeneration feature gate (which was enabled by default from Kubernetes v1.24 to v1.26), prevented Kubernetes from automatically creating these tokens for ServiceAccounts. The feature gate is removed in v1.27, because it was elevated to GA status; you can still create indefinite service account tokens manually, but should take into account the security implications.Note:For applications running outside your Kubernetes cluster, you might be considering creating a long-lived ServiceAccount token that is stored in a Secret. This allows authentication, but the Kubernetes project recommends you avoid this approach. Long-lived bearer tokens represent a security risk as, once disclosed, the token can be misused. Instead, consider using an alternative. For example, your external application can authenticate using a well-protected private key and a certificate, or using a custom mechanism such as an authentication webhook that you implement yourself.You can also use TokenRequest to obtain short-lived tokens for your external application.Restricting access to Secrets (deprecated)FEATURE STATE: Kubernetes v1.32 [deprecated]Note:kubernetes.io/enforce-mountable-secrets is deprecated since Kubernetes v1.32. Use separate namespaces to isolate access to mounted secrets.Kubernetes provides an annotation called kubernetes.io/enforce-mountable-secrets that you can add to your ServiceAccounts. When this annotation is applied, the ServiceAccount's secrets can only be mounted on specified types of resources, enhancing the security posture of your cluster.You can add the annotation to a ServiceAccount using a manifest:apiVersion: v1 kind: ServiceAccount metadata: annotations: kubernetes.io/enforce-mountable-secrets: "true" name: my-serviceaccount namespace: my-namespace When this annotation is set to "true", the Kubernetes control plane ensures that the Secrets from this ServiceAccount are subject to certain mounting restrictions.The name of each Secret that is mounted as a volume in a Pod must appear in the secrets field of the Pod's ServiceAccount.The name of each Secret referenced using envFrom in a Pod must also appear in the secrets field of the Pod's ServiceAccount.The name of each Secret referenced using imagePullSecrets in a Pod must also appear in the secrets field of the Pod's ServiceAccount.By understanding and enforcing these restrictions, cluster administrators can maintain a tighter security profile and ensure that secrets are accessed only by the appropriate resources.Authenticating service account credentialsServiceAccounts use signed JSON Web Tokens (JWTs) to authenticate to the Kubernetes API server, and to any other system where a trust relationship exists. Depending on how the token was issued (either time-limited using a TokenRequest or using a legacy mechanism with a Secret), a ServiceAccount token might also have an expiry time, an audience, and a time after which the token starts being valid. When a client that is acting as a ServiceAccount tries to communicate with the Kubernetes API server, the client includes an Authorization: Bearer <token> header with the HTTP request. The API server checks the validity of that bearer token as follows:Checks the token signature.Checks whether the token has expired.Checks whether object references in the token claims are currently valid.Checks whether the token is currently valid.Checks the audience claims.The TokenRequest API produces bound tokens for a ServiceAccount. This binding is linked to the lifetime of the client, such as a Pod, that is acting as that ServiceAccount. See Token Volume Projection for an example of a bound pod service account token's JWT schema and payload.For tokens issued using the TokenRequest API, the API server also checks that the specific object reference that is using the ServiceAccount still exists, matching by the unique ID of that object. For legacy tokens that are mounted as Secrets in Pods, the API server checks the token against the Secret.For more information about the authentication process, refer to Authentication.Authenticating service account credentials in your own codeIf you have services of your own that need to validate Kubernetes service account credentials, you can use the following methods:TokenReview API (recommended)OIDC discoveryThe Kubernetes project recommends that you use the TokenReview API, because this method invalidates tokens that are bound to API objects such as Secrets, ServiceAccounts, Pods or Nodes when those objects are deleted. For example, if you delete the Pod that contains a projected ServiceAccount token, the cluster invalidates that token immediately and a TokenReview immediately fails. If you use OIDC validation instead, your clients continue to treat the token as valid until the token reaches its expiration timestamp.Your application should always define the audience that it accepts, and should check that the token's audiences match the audiences that the application expects. This helps to minimize the scope of the token so that it can only be used in your application and nowhere else.AlternativesIssue your own tokens using another mechanism, and then use Webhook Token Authentication to validate bearer tokens using your own validation service.Provide your own identities to Pods.Use the SPIFFE CSI driver plugin to provide SPIFFE SVIDs as X.509 certificate pairs to Pods.🛇 This item links to a third party project or product that is not part of Kubernetes itself. More informationUse a service mesh such as Istio to provide certificates to Pods.Authenticate from outside the cluster to the API server without using service account tokens:Configure the API server to accept OpenID Connect (OIDC) tokens from your identity provider.Use service accounts or user accounts created using an external Identity and Access Management (IAM) service, such as from a cloud provider, to authenticate to your cluster.Use the CertificateSigningRequest API with client certificates.Configure the kubelet to retrieve credentials from an image registry.Use a Device Plugin to access a virtual Trusted Platform Module (TPM), which then allows authentication using a private key.What's nextLearn how to manage your ServiceAccounts as a cluster administrator.Learn how to assign a ServiceAccount to a Pod.Read the ServiceAccount API reference.Items on this page refer to third party products or projects that provide functionality required by Kubernetes. The Kubernetes project authors aren't responsible for those third-party products or projects. See the CNCF website guidelines for more details.You should read the content guide before proposing a change that adds an extra third-party link.