# Run a Replicated Stateful Application

Run a Replicated Stateful ApplicationThis page shows how to run a replicated stateful application using a StatefulSet. This application is a replicated MySQL database. The example topology has a single primary server and multiple replicas, using asynchronous row-based replication.Note:This is not a production configuration. MySQL settings remain on insecure defaults to keep the focus on general patterns for running stateful applications in Kubernetes.Before you beginYou need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. It is recommended to run this tutorial on a cluster with at least two nodes that are not acting as control plane hosts. If you do not already have a cluster, you can create one by using minikube or you can use one of these Kubernetes playgrounds:iximiuz LabsKillercodaKodeKloudPlay with KubernetesYou need to either have a dynamic PersistentVolume provisioner with a default StorageClass, or statically provision PersistentVolumes yourself to satisfy the PersistentVolumeClaims used here.This tutorial assumes you are familiar with PersistentVolumes and StatefulSets, as well as other core concepts like Pods, Services, and ConfigMaps.Some familiarity with MySQL helps, but this tutorial aims to present general patterns that should be useful for other systems.You are using the default namespace or another namespace that does not contain any conflicting objects.You need to have a AMD64-compatible CPU.ObjectivesDeploy a replicated MySQL topology with a StatefulSet.Send MySQL client traffic.Observe resistance to downtime.Scale the StatefulSet up and down.Deploy MySQLThe example MySQL deployment consists of a ConfigMap, two Services, and a StatefulSet.Create a ConfigMapCreate the ConfigMap from the following YAML configuration file:application/mysql/mysql-configmap.yaml apiVersion: v1 kind: ConfigMap metadata: name: mysql labels: app: mysql app.kubernetes.io/name: mysql data: primary.cnf: | # Apply this config only on the primary. [mysqld] log-bin replica.cnf: | # Apply this config only on replicas. [mysqld] super-read-only kubectl apply -f https://k8s.io/examples/application/mysql/mysql-configmap.yaml This ConfigMap provides my.cnf overrides that let you independently control configuration on the primary MySQL server and its replicas. In this case, you want the primary server to be able to serve replication logs to replicas and you want replicas to reject any writes that don't come via replication.There's nothing special about the ConfigMap itself that causes different portions to apply to different Pods. Each Pod decides which portion to look at as it's initializing, based on information provided by the StatefulSet controller.Create ServicesCreate the Services from the following YAML configuration file:application/mysql/mysql-services.yaml # Headless service for stable DNS entries of StatefulSet members. apiVersion: v1 kind: Service metadata: name: mysql labels: app: mysql app.kubernetes.io/name: mysql spec: ports: - name: mysql port: 3306 clusterIP: None selector: app: mysql --- # Client service for connecting to any MySQL instance for reads. # For writes, you must instead connect to the primary: mysql-0.mysql. apiVersion: v1 kind: Service metadata: name: mysql-read labels: app: mysql app.kubernetes.io/name: mysql readonly: "true" spec: ports: - name: mysql port: 3306 selector: app: mysql kubectl apply -f https://k8s.io/examples/application/mysql/mysql-services.yaml The headless Service provides a home for the DNS entries that the StatefulSet controllers creates for each Pod that's part of the set. Because the headless Service is named mysql, the Pods are accessible by resolving <pod-name>.mysql from within any other Pod in the same Kubernetes cluster and namespace.The client Service, called mysql-read, is a normal Service with its own cluster IP that distributes connections across all MySQL Pods that report being Ready. The set of potential endpoints includes the primary MySQL server and all replicas.Note that only read queries can use the load-balanced client Service. Because there is only one primary MySQL server, clients should connect directly to the primary MySQL Pod (through its DNS entry within the headless Service) to execute writes.Create the StatefulSetFinally, create the StatefulSet from the following YAML configuration file:application/mysql/mysql-statefulset.yaml apiVersion: apps/v1 kind: StatefulSet metadata: name: mysql spec: selector: matchLabels: app: mysql app.kubernetes.io/name: mysql serviceName: mysql replicas: 3 template: metadata: labels: app: mysql app.kubernetes.io/name: mysql spec: initContainers: - name: init-mysql image: mysql:5.7 command: - bash - "-c" - | set -ex # Generate mysql server-id from pod ordinal index. [[ $HOSTNAME =~ -([0-9]+)$ ]] || exit 1 ordinal=${BASH_REMATCH[1]} echo [mysqld] > /mnt/conf.d/server-id.cnf # Add an offset to avoid reserved server-id=0 value. echo server-id=$((100 + $ordinal)) >> /mnt/conf.d/server-id.cnf # Copy appropriate conf.d files from config-map to emptyDir. if [[ $ordinal -eq 0 ]]; then cp /mnt/config-map/primary.cnf /mnt/conf.d/ else cp /mnt/config-map/replica.cnf /mnt/conf.d/ fi volumeMounts: - name: conf mountPath: /mnt/conf.d - name: config-map mountPath: /mnt/config-map - name: clone-mysql image: gcr.io/google-samples/xtrabackup:1.0 command: - bash - "-c" - | set -ex # Skip the clone if data already exists. [[ -d /var/lib/mysql/mysql ]] && exit 0 # Skip the clone on primary (ordinal index 0). [[ `hostname` =~ -([0-9]+)$ ]] || exit 1 ordinal=${BASH_REMATCH[1]} [[ $ordinal -eq 0 ]] && exit 0 # Clone data from previous peer. ncat --recv-only mysql-$(($ordinal-1)).mysql 3307 | xbstream -x -C /var/lib/mysql # Prepare the backup. xtrabackup --prepare --target-dir=/var/lib/mysql volumeMounts: - name: data mountPath: /var/lib/mysql subPath: mysql - name: conf mountPath: /etc/mysql/conf.d containers: - name: mysql image: mysql:5.7 env: - name: MYSQL_ALLOW_EMPTY_PASSWORD value: "1" ports: - name: mysql containerPort: 3306 volumeMounts: - name: data mountPath: /var/lib/mysql subPath: mysql - name: conf mountPath: /etc/mysql/conf.d resources: requests: cpu: 500m memory: 1Gi livenessProbe: exec: command: ["mysqladmin", "ping"] initialDelaySeconds: 30 periodSeconds: 10 timeoutSeconds: 5 readinessProbe: exec: # Check we can execute queries over TCP (skip-networking is off). command: ["mysql", "-h", "127.0.0.1", "-e", "SELECT 1"] initialDelaySeconds: 5 periodSeconds: 2 timeoutSeconds: 1 - name: xtrabackup image: gcr.io/google-samples/xtrabackup:1.0 ports: - name: xtrabackup containerPort: 3307 command: - bash - "-c" - | set -ex cd /var/lib/mysql # Determine binlog position of cloned data, if any. if [[ -f xtrabackup_slave_info && "x$(<xtrabackup_slave_info)" != "x" ]]; then # XtraBackup already generated a partial "CHANGE MASTER TO" query # because we're cloning from an existing replica. (Need to remove the tailing semicolon!) cat xtrabackup_slave_info | sed -E 's/;$//g' > change_master_to.sql.in # Ignore xtrabackup_binlog_info in this case (it's useless). rm -f xtrabackup_slave_info xtrabackup_binlog_info elif [[ -f xtrabackup_binlog_info ]]; then # We're cloning directly from primary. Parse binlog position. [[ `cat xtrabackup_binlog_info` =~ ^(.*?)[[:space:]]+(.*?)$ ]] || exit 1 rm -f xtrabackup_binlog_info xtrabackup_slave_info echo "CHANGE MASTER TO MASTER_LOG_FILE='${BASH_REMATCH[1]}',\ MASTER_LOG_POS=${BASH_REMATCH[2]}" > change_master_to.sql.in fi # Check if we need to complete a clone by starting replication. if [[ -f change_master_to.sql.in ]]; then echo "Waiting for mysqld to be ready (accepting connections)" until mysql -h 127.0.0.1 -e "SELECT 1"; do sleep 1; done echo "Initializing replication from clone position" mysql -h 127.0.0.1 \ -e "$(<change_master_to.sql.in), \ MASTER_HOST='mysql-0.mysql', \ MASTER_USER='root', \ MASTER_PASSWORD='', \ MASTER_CONNECT_RETRY=10; \ START SLAVE;" || exit 1 # In case of container restart, attempt this at-most-once. mv change_master_to.sql.in change_master_to.sql.orig fi # Start a server to send backups when requested by peers. exec ncat --listen --keep-open --send-only --max-conns=1 3307 -c \ "xtrabackup --backup --slave-info --stream=xbstream --host=127.0.0.1 --user=root" volumeMounts: - name: data mountPath: /var/lib/mysql subPath: mysql - name: conf mountPath: /etc/mysql/conf.d resources: requests: cpu: 100m memory: 100Mi volumes: - name: conf emptyDir: {} - name: config-map configMap: name: mysql volumeClaimTemplates: - metadata: name: data spec: accessModes: ["ReadWriteOnce"] resources: requests: storage: 10Gi kubectl apply -f https://k8s.io/examples/application/mysql/mysql-statefulset.yaml You can watch the startup progress by running:kubectl get pods -l app=mysql --watch After a while, you should see all 3 Pods become Running:NAME READY STATUS RESTARTS AGE mysql-0 2/2 Running 0 2m mysql-1 2/2 Running 0 1m mysql-2 2/2 Running 0 1m Press Ctrl+C to cancel the watch.Note:If you don't see any progress, make sure you have a dynamic PersistentVolume provisioner enabled, as mentioned in the prerequisites.This manifest uses a variety of techniques for managing stateful Pods as part of a StatefulSet. The next section highlights some of these techniques to explain what happens as the StatefulSet creates Pods.Understanding stateful Pod initializationThe StatefulSet controller starts Pods one at a time, in order by their ordinal index. It waits until each Pod reports being Ready before starting the next one.In addition, the controller assigns each Pod a unique, stable name of the form <statefulset-name>-<ordinal-index>, which results in Pods named mysql-0, mysql-1, and mysql-2.The Pod template in the above StatefulSet manifest takes advantage of these properties to perform orderly startup of MySQL replication.Generating configurationBefore starting any of the containers in the Pod spec, the Pod first runs any init containers in the order defined.The first init container, named init-mysql, generates special MySQL config files based on the ordinal index.The script determines its own ordinal index by extracting it from the end of the Pod name, which is returned by the hostname command. Then it saves the ordinal (with a numeric offset to avoid reserved values) into a file called server-id.cnf in the MySQL conf.d directory. This translates the unique, stable identity provided by the StatefulSet into the domain of MySQL server IDs, which require the same properties.The script in the init-mysql container also applies either primary.cnf or replica.cnf from the ConfigMap by copying the contents into conf.d. Because the example topology consists of a single primary MySQL server and any number of replicas, the script assigns ordinal 0 to be the primary server, and everyone else to be replicas. Combined with the StatefulSet controller's deployment order guarantee, this ensures the primary MySQL server is Ready before creating replicas, so they can begin replicating.Cloning existing dataIn general, when a new Pod joins the set as a replica, it must assume the primary MySQL server might already have data on it. It also must assume that the replication logs might not go all the way back to the beginning of time. These conservative assumptions are the key to allow a running StatefulSet to scale up and down over time, rather than being fixed at its initial size.The second init container, named clone-mysql, performs a clone operation on a replica Pod the first time it starts up on an empty PersistentVolume. That means it copies all existing data from another running Pod, so its local state is consistent enough to begin replicating from the primary server.MySQL itself does not provide a mechanism to do this, so the example uses a popular open-source tool called Percona XtraBackup. During the clone, the source MySQL server might suffer reduced performance. To minimize impact on the primary MySQL server, the script instructs each Pod to clone from the Pod whose ordinal index is one lower. This works because the StatefulSet controller always ensures Pod N is Ready before starting Pod N+1.Starting replicationAfter the init containers complete successfully, the regular containers run. The MySQL Pods consist of a mysql container that runs the actual mysqld server, and an xtrabackup container that acts as a sidecar.The xtrabackup sidecar looks at the cloned data files and determines if it's necessary to initialize MySQL replication on the replica. If so, it waits for mysqld to be ready and then executes the CHANGE MASTER TO and START SLAVE commands with replication parameters extracted from the XtraBackup clone files.Once a replica begins replication, it remembers its primary MySQL server and reconnects automatically if the server restarts or the connection dies. Also, because replicas look for the primary server at its stable DNS name (mysql-0.mysql), they automatically find the primary server even if it gets a new Pod IP due to being rescheduled.Lastly, after starting replication, the xtrabackup container listens for connections from other Pods requesting a data clone. This server remains up indefinitely in case the StatefulSet scales up, or in case the next Pod loses its PersistentVolumeClaim and needs to redo the clone.Sending client trafficYou can send test queries to the primary MySQL server (hostname mysql-0.mysql) by running a temporary container with the mysql:5.7 image and running the mysql client binary.kubectl run mysql-client --image=mysql:5.7 -i --rm --restart=Never --\ mysql -h mysql-0.mysql <<EOF CREATE DATABASE test; CREATE TABLE test.messages (message VARCHAR(250)); INSERT INTO test.messages VALUES ('hello'); EOF Use the hostname mysql-read to send test queries to any server that reports being Ready:kubectl run mysql-client --image=mysql:5.7 -i -t --rm --restart=Never --\ mysql -h mysql-read -e "SELECT * FROM test.messages" You should get output like this:Waiting for pod default/mysql-client to be running, status is Pending, pod ready: false +---------+ | message | +---------+ | hello | +---------+ pod "mysql-client" deleted To demonstrate that the mysql-read Service distributes connections across servers, you can run SELECT @@server_id in a loop:kubectl run mysql-client-loop --image=mysql:5.7 -i -t --rm --restart=Never --\ bash -ic "while sleep 1; do mysql -h mysql-read -e 'SELECT @@server_id,NOW()'; done" You should see the reported @@server_id change randomly, because a different endpoint might be selected upon each connection attempt:+-------------+---------------------+ | @@server_id | NOW() | +-------------+---------------------+ | 100 | 2006-01-02 15:04:05 | +-------------+---------------------+ +-------------+---------------------+ | @@server_id | NOW() | +-------------+---------------------+ | 102 | 2006-01-02 15:04:06 | +-------------+---------------------+ +-------------+---------------------+ | @@server_id | NOW() | +-------------+---------------------+ | 101 | 2006-01-02 15:04:07 | +-------------+---------------------+ You can press Ctrl+C when you want to stop the loop, but it's useful to keep it running in another window so you can see the effects of the following steps.Simulate Pod and Node failureTo demonstrate the increased availability of reading from the pool of replicas instead of a single server, keep the SELECT @@server_id loop from above running while you force a Pod out of the Ready state.Break the Readiness probeThe readiness probe for the mysql container runs the command mysql -h 127.0.0.1 -e 'SELECT 1' to make sure the server is up and able to execute queries.One way to force this readiness probe to fail is to break that command:kubectl exec mysql-2 -c mysql -- mv /usr/bin/mysql /usr/bin/mysql.off This reaches into the actual container's filesystem for Pod mysql-2 and renames the mysql command so the readiness probe can't find it. After a few seconds, the Pod should report one of its containers as not Ready, which you can check by running:kubectl get pod mysql-2 Look for 1/2 in the READY column:NAME READY STATUS RESTARTS AGE mysql-2 1/2 Running 0 3m At this point, you should see your SELECT @@server_id loop continue to run, although it never reports 102 anymore. Recall that the init-mysql script defined server-id as 100 + $ordinal, so server ID 102 corresponds to Pod mysql-2.Now repair the Pod and it should reappear in the loop output after a few seconds:kubectl exec mysql-2 -c mysql -- mv /usr/bin/mysql.off /usr/bin/mysql Delete PodsThe StatefulSet also recreates Pods if they're deleted, similar to what a ReplicaSet does for stateless Pods.kubectl delete pod mysql-2 The StatefulSet controller notices that no mysql-2 Pod exists anymore, and creates a new one with the same name and linked to the same PersistentVolumeClaim. You should see server ID 102 disappear from the loop output for a while and then return on its own.Drain a NodeIf your Kubernetes cluster has multiple Nodes, you can simulate Node downtime (such as when Nodes are upgraded) by issuing a drain.First determine which Node one of the MySQL Pods is on:kubectl get pod mysql-2 -o wide The Node name should show up in the last column:NAME READY STATUS RESTARTS AGE IP NODE mysql-2 2/2 Running 0 15m 10.244.5.27 kubernetes-node-9l2t Then, drain the Node by running the following command, which cordons it so no new Pods may schedule there, and then evicts any existing Pods. Replace <node-name> with the name of the Node you found in the last step.Caution:Draining a Node can impact other workloads and applications running on the same node. Only perform the following step in a test cluster.# See above advice about impact on other workloads kubectl drain <node-name> --force --delete-emptydir-data --ignore-daemonsets Now you can watch as the Pod reschedules on a different Node:kubectl get pod mysql-2 -o wide --watch It should look something like this:NAME READY STATUS RESTARTS AGE IP NODE mysql-2 2/2 Terminating 0 15m 10.244.1.56 kubernetes-node-9l2t [...] mysql-2 0/2 Pending 0 0s <none> kubernetes-node-fjlm mysql-2 0/2 Init:0/2 0 0s <none> kubernetes-node-fjlm mysql-2 0/2 Init:1/2 0 20s 10.244.5.32 kubernetes-node-fjlm mysql-2 0/2 PodInitializing 0 21s 10.244.5.32 kubernetes-node-fjlm mysql-2 1/2 Running 0 22s 10.244.5.32 kubernetes-node-fjlm mysql-2 2/2 Running 0 30s 10.244.5.32 kubernetes-node-fjlm And again, you should see server ID 102 disappear from the SELECT @@server_id loop output for a while and then return.Now uncordon the Node to return it to a normal state:kubectl uncordon <node-name> Scaling the number of replicasWhen you use MySQL replication, you can scale your read query capacity by adding replicas. For a StatefulSet, you can achieve this with a single command:kubectl scale statefulset mysql --replicas=5 Watch the new Pods come up by running:kubectl get pods -l app=mysql --watch Once they're up, you should see server IDs 103 and 104 start appearing in the SELECT @@server_id loop output.You can also verify that these new servers have the data you added before they existed:kubectl run mysql-client --image=mysql:5.7 -i -t --rm --restart=Never --\ mysql -h mysql-3.mysql -e "SELECT * FROM test.messages" Waiting for pod default/mysql-client to be running, status is Pending, pod ready: false +---------+ | message | +---------+ | hello | +---------+ pod "mysql-client" deleted Scaling back down is also seamless:kubectl scale statefulset mysql --replicas=3 Note:Although scaling up creates new PersistentVolumeClaims automatically, scaling down does not automatically delete these PVCs.This gives you the choice to keep those initialized PVCs around to make scaling back up quicker, or to extract data before deleting them.You can see this by running:kubectl get pvc -l app=mysql Which shows that all 5 PVCs still exist, despite having scaled the StatefulSet down to 3:NAME STATUS VOLUME CAPACITY ACCESSMODES AGE data-mysql-0 Bound pvc-8acbf5dc-b103-11e6-93fa-42010a800002 10Gi RWO 20m data-mysql-1 Bound pvc-8ad39820-b103-11e6-93fa-42010a800002 10Gi RWO 20m data-mysql-2 Bound pvc-8ad69a6d-b103-11e6-93fa-42010a800002 10Gi RWO 20m data-mysql-3 Bound pvc-50043c45-b1c5-11e6-93fa-42010a800002 10Gi RWO 2m data-mysql-4 Bound pvc-500a9957-b1c5-11e6-93fa-42010a800002 10Gi RWO 2m If you don't intend to reuse the extra PVCs, you can delete them:kubectl delete pvc data-mysql-3 kubectl delete pvc data-mysql-4 Cleaning upCancel the SELECT @@server_id loop by pressing Ctrl+C in its terminal, or running the following from another terminal:kubectl delete pod mysql-client-loop --now Delete the StatefulSet. This also begins terminating the Pods.kubectl delete statefulset mysql Verify that the Pods disappear. They might take some time to finish terminating.kubectl get pods -l app=mysql You'll know the Pods have terminated when the above returns:No resources found. Delete the ConfigMap, Services, and PersistentVolumeClaims.kubectl delete configmap,service,pvc -l app=mysql If you manually provisioned PersistentVolumes, you also need to manually delete them, as well as release the underlying resources. If you used a dynamic provisioner, it automatically deletes the PersistentVolumes when it sees that you deleted the PersistentVolumeClaims. Some dynamic provisioners (such as those for EBS and PD) also release the underlying resources upon deleting the PersistentVolumes.What's nextLearn more about scaling a StatefulSet.Learn more about debugging a StatefulSet.Learn more about deleting a StatefulSet.Learn more about force deleting StatefulSet Pods.Look in the Helm Charts repository for other stateful application examples.


# Kubectl user preferences (kuberc)

Kubectl user preferences (kuberc)FEATURE STATE: Kubernetes 1.34 [beta]A Kubernetes kuberc configuration file allows you to define preferences for kubectl, such as default options and command aliases. Unlike the kubeconfig file, a kuberc configuration file does not contain cluster details, usernames or passwords.The default location of this configuration file is $HOME/.kube/kuberc. To provide kubectl with a path to a custom kuberc file, use the --kuberc command line option, or set the KUBERC environment variable.A kuberc using the kubectl.config.k8s.io/v1beta1 format allows you to define two types of user preferences:Aliases - allow you to create shorter versions of your favorite commands, optionally setting options and arguments.Defaults - allow you to configure default option values for your favorite commands.aliasesWithin a kuberc configuration, the aliases section allows you to define custom shortcuts for kubectl commands, optionally with preset command line arguments and flags.This


# CRI Pod & Container Metrics

CRI Pod & Container MetricsCollection of Pod & Container metrics via the CRI.FEATURE STATE: Kubernetes v1.23 [alpha]The kubelet collects pod and container metrics via cAdvisor. As an alpha feature, Kubernetes lets you configure the collection of pod and container metrics via the Container Runtime Interface (CRI). You must enable the PodAndContainerStatsFromCRI feature gate and use a compatible CRI implementation (containerd >= 1.6.0, CRI-O >= 1.23.0) to use the CRI based collection mechanism.CRI Pod & Container MetricsWith PodAndContainerStatsFromCRI enabled, the kubelet polls the underlying container runtime for pod and container stats instead of inspecting the host system directly using cAdvisor. The benefits of relying on the container runtime for this information as opposed to direct collection with cAdvisor include:Potential improved performance if the container runtime already collects this information during normal operations. In this case, the data can be re-used instead of being aggregated again by the kubelet.It further decouples the kubelet and the container runtime allowing collection of metrics for container runtimes that don't run processes directly on the host with kubelet where they are observable by cAdvisor (for example: container runtimes that use virtualization).


# kubectl config unset

kubectl config unsetSynopsisUnset an individual value in a kubeconfig file.PROPERTY_NAME is a dot delimited name where each token represents either an attribute name or a map key. Map keys may not contain dots.kubectl config unset PROPERTY_NAME Examples # Unset the current-context kubectl config unset current-context # Unset namespace in foo context kubectl config unset contexts.foo.namespace Options-h, --helphelp for unsetParent Options Inherited--as stringUsername to impersonate for the operation. User could be a regular user or a service account in a namespace.--as-group stringsGroup to impersonate for the operation, this flag can be repeated to specify multiple groups.--as-uid stringUID to impersonate for the operation.--cache-dir string Default: "$HOME/.kube/cache"Default cache directory--certificate-authority stringPath to a cert file for the certificate authority--client-certificate stringPath to a client certificate file for TLS--client-key stringPath to a client key file for TLS--cluster stringThe name of the kubeconfig cluster to use--context stringThe name of the kubeconfig context to use--disable-compressionIf true, opt-out of response compression for all requests to the server--insecure-skip-tls-verifyIf true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure--kubeconfig stringuse a particular kubeconfig file--kuberc stringPath to the kuberc file to use for preferences. This can be disabled by exporting KUBECTL_KUBERC=false feature gate or turning off the feature KUBERC=off.--match-server-versionRequire server version to match client version-n, --namespace stringIf present, the namespace scope for this CLI request--password stringPassword for basic authentication to the API server--profile string Default: "none"Name of profile to capture. One of (none|cpu|heap|goroutine|threadcreate|block|mutex)--profile-output string Default: "profile.pprof"Name of the file to write the profile to--request-timeout string Default: "0"The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests.-s, --server stringThe address and port of the Kubernetes API server--storage-driver-buffer-duration duration Default: 1m0sWrites in the storage driver will be buffered for this duration, and committed to the non memory backends as a single transaction--storage-driver-db string Default: "cadvisor"database name--storage-driver-host string Default: "localhost:8086"database host:port--storage-driver-password string Default: "root"database password--storage-driver-secureuse secure connection with database--storage-driver-table string Default: "stats"table name--storage-driver-user string Default: "root"database username--tls-server-name stringServer name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used--token stringBearer token for authentication to the API server--user stringThe name of the kubeconfig user to use--username stringUsername for basic authentication to the API server--version version[=true]--version, --version=raw prints version information and quits; --version=vX.Y.Z... sets the reported version--warnings-as-errorsTreat warnings received from the server as errors and exit with a non-zero exit codeSee Alsokubectl config - Modify kubeconfig filesThis page is automatically generated.If you plan to report an issue with this page, mention that the page is auto-generated in your issue description. The fix may need to happen elsewhere in the Kubernetes project.


# Object Names and IDs

Object Names and IDsEach object in your cluster has a Name that is unique for that type of resource. Every Kubernetes object also has a UID that is unique across your whole cluster.For example, you can only have one Pod named myapp-1234 within the same namespace, but you can have one Pod and one Deployment that are each named myapp-1234.For non-unique user-provided attributes, Kubernetes provides labels and annotations.NamesA client-provided string that refers to an object in a resource URL, such as /api/v1/pods/some-name.Only one object of a given kind can have a given name at a time. However, if you delete the object, you can make a new object with the same name.Names must be unique across all API versions of the same resource. API resources are distinguished by their API group, resource type, namespace (for namespaced resources), and name. In other words, API version is irrelevant in this context.Note:In cases when objects represent a physical entity, like a Node representing a physical host, when the host is re-created under the same name without deleting and re-creating the Node, Kubernetes treats the new host as the old one, which may lead to inconsistencies.The server may generate a name when generateName is provided instead of name in a resource create request. When generateName is used, the provided value is used as a name prefix, which server appends a generated suffix to. Even though the name is generated, it may conflict with existing names resulting in a HTTP 409 response. This became far less likely to happen in Kubernetes v1.31 and later, since the server will make up to 8 attempt to generate a unique name before returning a HTTP 409 response.Below are four types of commonly used name constraints for resources.DNS Subdomain NamesMost resource types require a name that can be used as a DNS subdomain name as defined in RFC 1123. This means the name must:contain no more than 253 characterscontain only lowercase alphanumeric characters, '-' or '.'start with an alphanumeric characterend with an alphanumeric characterRFC 1123 Label NamesSome resource types require their names to follow the DNS label standard as defined in RFC 1123. This means the name must:contain at most 63 characterscontain only lowercase alphanumeric characters or '-'start with an alphanumeric characterend with an alphanumeric characterRFC 1035 Label NamesSome resource types require their names to follow the DNS label standard as defined in RFC 1035. This means the name must:contain at most 63 characterscontain only lowercase alphanumeric characters or '-'start with an alphabetic characterend with an alphanumeric characterNote:The only difference between the RFC 1035 and RFC 1123 label standards is that RFC 1123 labels are allowed to start with a digit, whereas RFC 1035 labels can start with a lowercase alphabetic character only.Path Segment NamesSome resource types require their names to be able to be safely encoded as a path segment. In other words, the name may not be "." or ".." and the name may not contain "/" or "%".Here's an example manifest for a Pod named nginx-demo.apiVersion: v1 kind: Pod metadata: name: nginx-demo spec: containers: - name: nginx image: nginx:1.14.2 ports: - containerPort: 80 Note:Some resource types have additional restrictions on their names.UIDsA Kubernetes systems-generated string to uniquely identify objects.Every object created over the whole lifetime of a Kubernetes cluster has a distinct UID. It is intended to distinguish between historical occurrences of similar entities.Kubernetes UIDs are universally unique identifiers (also known as UUIDs). UUIDs are standardized as ISO/IEC 9834-8 and as ITU-T X.667.What's nextRead about labels and annotations in Kubernetes.See the Identifiers and Names in Kubernetes design document.


# kubectl config use-context

kubectl config use-contextSynopsisSet the current-context in a kubeconfig file.kubectl config use-context CONTEXT_NAME Examples # Use the context for the minikube cluster kubectl config use-context minikube Options-h, --helphelp for use-contextParent Options Inherited--as stringUsername to impersonate for the operation. User could be a regular user or a service account in a namespace.--as-group stringsGroup to impersonate for the operation, this flag can be repeated to specify multiple groups.--as-uid stringUID to impersonate for the operation.--cache-dir string Default: "$HOME/.kube/cache"Default cache directory--certificate-authority stringPath to a cert file for the certificate authority--client-certificate stringPath to a client certificate file for TLS--client-key stringPath to a client key file for TLS--cluster stringThe name of the kubeconfig cluster to use--context stringThe name of the kubeconfig context to use--disable-compressionIf true, opt-out of response compression for all requests to the server--insecure-skip-tls-verifyIf true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure--kubeconfig stringuse a particular kubeconfig file--kuberc stringPath to the kuberc file to use for preferences. This can be disabled by exporting KUBECTL_KUBERC=false feature gate or turning off the feature KUBERC=off.--match-server-versionRequire server version to match client version-n, --namespace stringIf present, the namespace scope for this CLI request--password stringPassword for basic authentication to the API server--profile string Default: "none"Name of profile to capture. One of (none|cpu|heap|goroutine|threadcreate|block|mutex)--profile-output string Default: "profile.pprof"Name of the file to write the profile to--request-timeout string Default: "0"The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests.-s, --server stringThe address and port of the Kubernetes API server--storage-driver-buffer-duration duration Default: 1m0sWrites in the storage driver will be buffered for this duration, and committed to the non memory backends as a single transaction--storage-driver-db string Default: "cadvisor"database name--storage-driver-host string Default: "localhost:8086"database host:port--storage-driver-password string Default: "root"database password--storage-driver-secureuse secure connection with database--storage-driver-table string Default: "stats"table name--storage-driver-user string Default: "root"database username--tls-server-name stringServer name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used--token stringBearer token for authentication to the API server--user stringThe name of the kubeconfig user to use--username stringUsername for basic authentication to the API server--version version[=true]--version, --version=raw prints version information and quits; --version=vX.Y.Z... sets the reported version--warnings-as-errorsTreat warnings received from the server as errors and exit with a non-zero exit codeSee Alsokubectl config - Modify kubeconfig filesThis page is automatically generated.If you plan to report an issue with this page, mention that the page is auto-generated in your issue description. The fix may need to happen elsewhere in the Kubernetes project.


# Enable Or Disable A Kubernetes API

Enable Or Disable A Kubernetes APIThis page shows how to enable or disable an API version from your cluster's control plane.Specific API versions can be turned on or off by passing --runtime-config=api/<version> as a command line argument to the API server. The values for this argument are a comma-separated list of API versions. Later values override earlier values.The runtime-config command line argument also supports 2 special keys:api/all, representing all known APIsapi/legacy, representing only legacy APIs. Legacy APIs are any APIs that have been explicitly deprecated.For example, to turn off all API versions except v1, pass --runtime-config=api/all=false,api/v1=true to the kube-apiserver.What's nextRead the full documentation for the kube-apiserver component.


# Authenticating

AuthenticatingThis page provides an overview of authentication in Kubernetes, with a focus on authentication to the Kubernetes API.Users in KubernetesAll Kubernetes clusters have two categories of users: service accounts managed by Kubernetes, and normal users.It is assumed that a cluster-independent service manages normal users in the following ways:an administrator distributing private keysa user store like Keystone or Google Accountsa file with a list of usernames and passwordsIn this regard, Kubernetes does not have objects which represent normal user accounts. Normal users cannot be added to a cluster through an API call.Even though a normal user cannot be added via an API call, any user that presents a valid certificate signed by the cluster's certificate authority (CA) is considered authenticated. In this configuration, Kubernetes determines the username from the common name field in the 'subject' of the cert (e.g., "/CN=bob"). From there, the role based access control (RBAC) sub-system would determine whether the user is authorized to perform a specific operation on a resource.In contrast, service accounts are users managed by the Kubernetes API. They are bound to specific namespaces, and created automatically by the API server or manually through API calls. Service accounts are tied to a set of credentials stored as Secrets, which are mounted into pods allowing in-cluster processes to talk to the Kubernetes API.API requests are tied to either a normal user or a service account, or are treated as anonymous requests. This means every process inside or outside the cluster, from a human user typing kubectl on a workstation, to kubelets on nodes, to members of the control plane, must authenticate when making requests to the API server, or be treated as an anonymous user.Authentication strategiesKubernetes uses client certificates, bearer tokens, or an authenticating proxy to authenticate API requests through authentication plugins. As HTTP requests are made to the API server, plugins attempt to associate the following attributes with the request:Username: a string which identifies the end user. Common values might be kube-admin or jane@example.com.UID: a string which identifies the end user and attempts to be more consistent and unique than username.Groups: a set of strings, each of which indicates the user's membership in a named logical collection of users. Common values might be system:masters or devops-team.Extra fields: a map of strings to list of strings which holds additional information authorizers may find useful.Note:All values are opaque to the authentication system and only hold significance when interpreted by an authorizer.Authentication methodsYou can enable multiple authentication methods at once. You should usually use at least two methods:service account tokens for service accountsat least one other method for user authentication.When multiple authenticator modules are enabled, the first module to successfully authenticate the request short-circuits evaluation. The API server does not guarantee the order authenticators run in.The system:authenticated group is included in the list of groups for all authenticated users.Integrations with other authentication protocols (LDAP, SAML, Kerberos, alternate x509 schemes, etc) are available; for example using an authenticating proxy or the authentication webhook.X.509 client certificatesClient certificate authentication is enabled by passing the --client-ca-file=SOMEFILE option to API server. The referenced file must contain one or more certificate authorities to use to validate client certificates presented to the API server. If a client certificate is presented and verified, the common name of the subject is used as the user name for the request. As of Kubernetes 1.4, client certificates can also indicate a user's group memberships using the certificate's organization fields. To include multiple group memberships for a user, include multiple organization fields in the certificate.For example, using the openssl command line tool to generate a certificate signing request:openssl req -new -key jbeda.pem -out jbeda-csr.pem -subj "/CN=jbeda/O=app1/O=app2" This would create a CSR for the username "jbeda", belonging to two groups, "app1" and "app2".See Managing Certificates for how to generate a client cert.Putting a bearer token in a requestWhen using bearer token authentication from an http client, the API server expects an Authorization header with a value of Bearer <token>. The bearer token must be a character sequence that can be put in an HTTP header value using no more than the encoding and quoting facilities of HTTP. For example: if the bearer token is 31ada4fd-adec-460c-809a-9e56ceb75269 then it would appear in an HTTP header as shown below.Authorization: Bearer 31ada4fd-adec-460c-809a-9e56ceb75269 Bootstrap tokensFEATURE STATE: Kubernetes v1.18 [stable]To allow for streamlined bootstrapping for new clusters, Kubernetes includes a dynamically-managed Bearer token type called a Bootstrap Token. These tokens are stored as Secrets in the kube-system namespace, where they can be dynamically managed and created. Controller Manager contains a TokenCleaner controller that deletes bootstrap tokens as they expire.The tokens are of the form [a-z0-9]{6}.[a-z0-9]{16}. The first component is a Token ID and the second component is the Token Secret. You specify the token in an HTTP header as follows:Authorization: Bearer 781292.db7bc3a58fc5f07e You must enable the Bootstrap Token Authenticator with the --enable-bootstrap-token-auth flag on the API Server. You must enable the TokenCleaner controller via the --controllers command line argument for kube-controller-manager. This is done with something like --controllers=*,tokencleaner. The kubeadm tool will do this for you if you are using it to bootstrap a cluster.The authenticator authenticates as system:bootstrap:<Token ID>. It is included in the system:bootstrappers group. The naming and groups are intentionally limited to discourage users from using these tokens past bootstrapping. The user names and group can be used (and are used by kubeadm) to craft the appropriate authorization policies to support bootstrapping a cluster.Please see Bootstrap Tokens for in depth documentation on the Bootstrap Token authenticator and controllers along with how to manage these tokens with kubeadm.Service account tokensA service account is an automatically enabled authenticator that uses signed bearer tokens to verify requests. The plugin takes two optional flags:--service-account-key-file File containing PEM-encoded x509 RSA or ECDSA private or public keys, used to verify ServiceAccount tokens. The specified file can contain multiple keys, and the flag can be specified multiple times with different files. If unspecified, --tls-private-key-file is used.--service-account-lookup If enabled, tokens which are deleted from the API will be revoked.Service accounts are usually created automatically by the API server and associated with pods running in the cluster through the ServiceAccount Admission Controller. Bearer tokens are mounted into pods at well-known locations, and allow in-cluster processes to talk to the API server. Accounts may be explicitly associated with pods using the serviceAccountName field of a PodSpec.Note:serviceAccountName is usually omitted because this is done automatically.apiVersion: apps/v1 # this apiVersion is relevant as of Kubernetes 1.9 kind: Deployment metadata: name: nginx-deployment namespace: default spec: replicas: 3 template: metadata: # ... spec: serviceAccountName: bob-the-bot containers: - name: nginx image: nginx:1.14.2 Service account bearer tokens are perfectly valid to use outside the cluster and can be used to create identities for long standing jobs that wish to talk to the Kubernetes API. To manually create a service account, use the kubectl create serviceaccount (NAME) command. This creates a service account in the current namespace.kubectl create serviceaccount jenkins serviceaccount/jenkins created You can manually create an associated token:kubectl create token jenkins eyJhbGciOiJSUzI1NiIsImtp... The created token is a signed JSON Web Token (JWT).The signed JWT can be used as a bearer token to authenticate as the given service account. See above for how the token is included in a request. Normally these tokens are mounted into pods for in-cluster access to the API server, but can be used from outside the cluster as well.Service accounts authenticate with the username system:serviceaccount:(NAMESPACE):(SERVICEACCOUNT), and are assigned to the groups system:serviceaccounts and system:serviceaccounts:(NAMESPACE).Warning:Because service account tokens can also be stored in Secret API objects, any user with write access to Secrets can request a token, and any user with read access to those Secrets can authenticate as the service account. Be cautious when granting permissions to service accounts and read or write capabilities for Secrets.External integrationsKubernetes has native support for OpenID Connect (OIDC); see OpenID Connect tokens.Integrations with other authentication protocols (for example: LDAP, SAML, Kerberos, alternate X.509 schemes) can be accomplished using an authenticating proxy or by integrating with an authentication webhook.You can also use any custom method that issues client X.509 certificates to clients, provided that the API server will trust the valid certificates. Read X.509 client certificates to learn about how to generate a certificate.If you do issue certificates to clients, it is up to you (as a cloud platform administrator) to make sure that the certificate validity period, and other design choices you make, provide a suitable level of security.OpenID Connect tokensOpenID Connect is a flavor of OAuth2 supported by some OAuth2 providers, notably Microsoft Entra ID, Salesforce, and Google. The protocol's main extension of OAuth2 is an additional field returned with the access token called an ID Token. This token is a JSON Web Token (JWT) with well known fields, such as a user's email, signed by the server.To identify the user, the authenticator uses the id_token (not the access_token) from the OAuth2 token response as a bearer token. See above for how the token is included in a request.sequenceDiagram participant user as User participant idp as Identity Provider participant kube as kubectl participant api as API Server user ->> idp: 1. Log in to IdP activate idp idp -->> user: 2. Provide access_token,id_token, and refresh_token deactivate idp activate user user ->> kube: 3. Call kubectlwith --token being the id_tokenOR add tokens to .kube/config deactivate user activate kube kube ->> api: 4. Authorization: Bearer... deactivate kube activate api api ->> api: 5. Is JWT signature valid? api ->> api: 6. Has the JWT expired? (iat+exp) api ->> api: 7. User authorized? api -->> kube: 8. Authorized: Performaction and return result deactivate api activate kube kube --x user: 9. Return result deactivate kubeJavaScript must be enabled to view this contentLog in to your identity providerYour identity provider will provide you with an access_token, id_token and a refresh_tokenWhen using kubectl, use your id_token with the --token command line argument or add it directly to your kubeconfigkubectl sends your id_token in a header called Authorization to the API serverThe API server will make sure the JWT signature is validCheck to make sure the id_token hasn't expiredPerform claim and/or user validation if CEL expressions are configured with AuthenticationConfiguration.Make sure the user is authorizedOnce authorized the API server returns a response to kubectlkubectl provides


# Configure GMSA for Windows Pods and containers

Configure GMSA for Windows Pods and containersFEATURE STATE: Kubernetes v1.18 [stable]This page shows how to configure Group Managed Service Accounts (GMSA) for Pods and containers that will run on Windows nodes. Group Managed Service Accounts are a specific type of Active Directory account that provides automatic password management, simplified service principal name (SPN) management, and the ability to delegate the management to other administrators across multiple servers.In Kubernetes, GMSA credential specs are configured at a Kubernetes cluster-wide scope as Custom Resources. Windows Pods, as well as individual containers within a Pod, can be configured to use a GMSA for domain based functions (e.g. Kerberos authentication) when interacting with other Windows services.Before you beginYou need to have a Kubernetes cluster and the kubectl command-line tool must be configured to communicate with your cluster. The cluster is expected to have Windows worker nodes. This section covers a set of initial steps required once for each cluster:Install the GMSACredentialSpec CRDA CustomResourceDefinition(CRD) for GMSA credential spec resources needs to be configured on the cluster to define the custom resource type GMSACredentialSpec. Download the GMSA CRD YAML and save it as gmsa-crd.yaml. Next, install the CRD with kubectl apply -f gmsa-crd.yamlInstall webhooks to validate GMSA usersTwo webhooks need to be configured on the Kubernetes cluster to populate and validate GMSA credential spec references at the Pod or container level:A mutating webhook that expands references to GMSAs (by name from a Pod specification) into the full credential spec in JSON form within the Pod spec.A validating webhook ensures all references to GMSAs are authorized to be used by the Pod service account.Installing the above webhooks and associated objects require the steps below:Create a certificate key pair (that will be used to allow the webhook container to communicate to the cluster)Install a secret with the certificate from above.Create a deployment for the core webhook logic.Create the validating and mutating webhook configurations referring to the deployment.A script can be used to deploy and configure the GMSA webhooks and associated objects mentioned above. The script can be run with a --dry-run=server option to allow you to review the changes that would be made to your cluster.The YAML template used by the script may also be used to deploy the webhooks and associated objects manually (with appropriate substitutions for the parameters)Configure GMSAs and Windows nodes in Active DirectoryBefore Pods in Kubernetes can be configured to use GMSAs, the desired GMSAs need to be provisioned in Active Directory as described in the Windows GMSA documentation. Windows worker nodes (that are part of the Kubernetes cluster) need to be configured in Active Directory to access the secret credentials associated with the desired GMSA as described in the Windows GMSA documentation.Create GMSA credential spec resourcesWith the GMSACredentialSpec CRD installed (as described earlier), custom resources containing GMSA credential specs can be configured. The GMSA credential spec does not contain secret or sensitive data. It is information that a container runtime can use to describe the desired GMSA of a container to Windows. GMSA credential specs can be generated in YAML format with a utility PowerShell script.Following are the steps for generating a GMSA credential spec YAML manually in JSON format and then converting it:Import the CredentialSpec module: ipmo CredentialSpec.psm1Create a credential spec in JSON format using New-CredentialSpec. To create a GMSA credential spec named WebApp1, invoke New-CredentialSpec -Name WebApp1 -AccountName WebApp1 -Domain $(Get-ADDomain -Current LocalComputer)Use Get-CredentialSpec to show the path of the JSON file.Convert the credspec file from JSON to YAML format and apply the necessary header fields apiVersion, kind, metadata and credspec to make it a GMSACredentialSpec custom resource that can be configured in Kubernetes.The following YAML configuration describes a GMSA credential spec named gmsa-WebApp1:apiVersion: windows.k8s.io/v1 kind: GMSACredentialSpec metadata: name: gmsa-WebApp1 # This is an arbitrary name but it will be used as a reference credspec: ActiveDirectoryConfig: GroupManagedServiceAccounts: - Name: WebApp1 # Username of the GMSA account Scope: CONTOSO # NETBIOS Domain Name - Name: WebApp1 # Username of the GMSA account Scope: contoso.com # DNS Domain Name CmsPlugins: - ActiveDirectory DomainJoinConfig: DnsName: contoso.com # DNS Domain Name DnsTreeName: contoso.com # DNS Domain Name Root Guid: 244818ae-87ac-4fcd-92ec-e79e5252348a # GUID of the Domain MachineAccountName: WebApp1 # Username of the GMSA account NetBiosName: CONTOSO # NETBIOS Domain Name Sid: S-1-5-21-2126449477-2524075714-3094792973 # SID of the Domain The above credential spec resource may be saved as gmsa-Webapp1-credspec.yaml and applied to the cluster using: kubectl apply -f gmsa-Webapp1-credspec.ymlConfigure cluster role to enable RBAC on specific GMSA credential specsA cluster role needs to be defined for each GMSA credential spec resource. This authorizes the use verb on a specific GMSA resource by a subject which is typically a service account. The following example shows a cluster role that authorizes usage of the gmsa-WebApp1 credential spec from above. Save the file as gmsa-webapp1-role.yaml and apply using kubectl apply -f gmsa-webapp1-role.yaml# Create the Role to read the credspec apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: webapp1-role rules: - apiGroups: ["windows.k8s.io"] resources: ["gmsacredentialspecs"] verbs: ["use"] resourceNames: ["gmsa-WebApp1"] Assign role to service accounts to use specific GMSA credspecsA service account (that Pods will be configured with) needs to be bound to the cluster role create above. This authorizes the service account to use the desired GMSA credential spec resource. The following shows the default service account being bound to a cluster role webapp1-role to use gmsa-WebApp1 credential spec resource created above.apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: allow-default-svc-account-read-on-gmsa-WebApp1 namespace: default subjects: - kind: ServiceAccount name: default namespace: default roleRef: kind: ClusterRole name: webapp1-role apiGroup: rbac.authorization.k8s.io Configure GMSA credential spec reference in Pod specThe Pod spec field securityContext.windowsOptions.gmsaCredentialSpecName is used to specify references to desired GMSA credential spec custom resources in Pod specs. This configures all containers in the Pod spec to use the specified GMSA. A sample Pod spec with the annotation populated to refer to gmsa-WebApp1:apiVersion: apps/v1 kind: Deployment metadata: labels: run: with-creds name: with-creds namespace: default spec: replicas: 1 selector: matchLabels: run: with-creds template: metadata: labels: run: with-creds spec: securityContext: windowsOptions: gmsaCredentialSpecName: gmsa-webapp1 containers: - image: mcr.microsoft.com/windows/servercore/iis:windowsservercore-ltsc2019 imagePullPolicy: Always name: iis nodeSelector: kubernetes.io/os: windows Individual containers in a Pod spec can also specify the desired GMSA credspec using a per-container securityContext.windowsOptions.gmsaCredentialSpecName field. For example:apiVersion: apps/v1 kind: Deployment metadata: labels: run: with-creds name: with-creds namespace: default spec: replicas: 1 selector: matchLabels: run: with-creds template: metadata: labels: run: with-creds spec: containers: - image: mcr.microsoft.com/windows/servercore/iis:windowsservercore-ltsc2019 imagePullPolicy: Always name: iis securityContext: windowsOptions: gmsaCredentialSpecName: gmsa-Webapp1 nodeSelector: kubernetes.io/os: windows As Pod specs with GMSA fields populated (as described above) are applied in a cluster, the following sequence of events take place:The mutating webhook resolves and expands all references to GMSA credential spec resources to the contents of the GMSA credential spec.The validating webhook ensures the service account associated with the Pod is authorized for the use verb on the specified GMSA credential spec.The container runtime configures each Windows container with the specified GMSA credential spec so that the container can assume the identity of the GMSA in Active Directory and access services in the domain using that identity.Authenticating to network shares using hostname or FQDNIf you are experiencing issues connecting to SMB shares from Pods using hostname or FQDN, but are able to access the shares via their IPv4 address then make sure the following registry key is set on the Windows nodes.reg add "HKLM\SYSTEM\CurrentControlSet\Services\hns\State" /v EnableCompartmentNamespace /t REG_DWORD /d 1 Running Pods will then need to be recreated to pick up the behavior changes. More information on how this registry key is used can be found hereTroubleshootingIf you are having difficulties getting GMSA to work in your environment, there are a few troubleshooting steps you can take.First, make sure the credspec has been passed to the Pod. To do this you will need to exec into one of your Pods and check the output of the nltest.exe /parentdomain command.In the example below the Pod did not get the credspec correctly:kubectl exec -it iis-auth-7776966999-n5nzr powershell.exe nltest.exe /parentdomain results in the following error:Getting parent domain failed: Status = 1722 0x6ba RPC_S_SERVER_UNAVAILABLE If your Pod did get the credspec correctly, then next check communication with the domain. First, from inside of your Pod, quickly do an nslookup to find the root of your domain.This will tell us 3 things:The Pod can reach the DCThe DC can reach the PodDNS is working correctly.If the DNS and communication test passes, next you will need to check if the Pod has established secure channel communication with the domain. To do this, again, exec into your Pod and run the nltest.exe /query command.nltest.exe /query Results in the following output:I_NetLogonControl failed: Status = 1722 0x6ba RPC_S_SERVER_UNAVAILABLE This tells us that for some reason, the Pod was unable to logon to the domain using the account specified in the credspec. You can try to repair the secure channel by running the following:nltest /sc_reset:domain.example If the command is successful you will see and output similar to this:Flags: 30 HAS_IP HAS_TIMESERV Trusted DC Name \\dc10.domain.example Trusted DC Connection Status Status = 0 0x0 NERR_Success The command completed successfully If the above corrects the error, you can automate the step by adding the following lifecycle hook to your Pod spec. If it did not correct the error, you will need to examine your credspec again and confirm that it is correct and complete. image: registry.domain.example/iis-auth:1809v1 lifecycle: postStart: exec: command: ["powershell.exe","-command","do { Restart-Service -Name netlogon } while ( $($Result = (nltest.exe /query); if ($Result -like '*0x0 NERR_Success*') {return $true} else {return $false}) -eq $false)"] imagePullPolicy: IfNotPresent If you add the lifecycle section show above to your Pod spec, the Pod will execute the commands listed to restart the netlogon service until the nltest.exe /query command exits without error.


# Kubernetes Security and Disclosure Information

Kubernetes Security and Disclosure InformationThis page describes Kubernetes security and disclosure information.Security AnnouncementsJoin the kubernetes-security-announce group for emails about security and major API announcements.Report a VulnerabilityWe're extremely grateful for security researchers and users that report vulnerabilities to the Kubernetes Open Source Community. All reports are thoroughly investigated by a set of community volunteers.To make a report, submit your vulnerability to the Kubernetes bug bounty program. This allows triage and handling of the vulnerability with standardized response times.You can also email the private security@kubernetes.io list with the security details and the details expected for all Kubernetes bug reports.You may encrypt your email to this list using the GPG keys of the Security Response Committee members. Encryption using GPG is NOT required to make a disclosure.When Should I Report a Vulnerability?You think you discovered a potential security vulnerability in KubernetesYou are unsure how a vulnerability affects KubernetesYou think you discovered a vulnerability in another project that Kubernetes depends onFor projects with their own vulnerability reporting and disclosure process, please report it directly thereWhen Should I NOT Report a Vulnerability?You need help tuning Kubernetes components for securityYou need help applying security related updatesYour issue is not security relatedSecurity Vulnerability ResponseEach report is acknowledged and analyzed by Security Response Committee members within 3 working days. This will set off the Security Release Process.Any vulnerability information shared with Security Response Committee stays within Kubernetes project and will not be disseminated to other projects unless it is necessary to get the issue fixed.As the security issue moves from triage, to identified fix, to release planning we will keep the reporter updated.Public Disclosure TimingA public disclosure date is negotiated by the Kubernetes Security Response Committee and the bug submitter. We prefer to fully disclose the bug as soon as possible once a user mitigation is available. It is reasonable to delay disclosure when the bug or the fix is not yet fully understood, the solution is not well-tested, or for vendor coordination. The timeframe for disclosure is from immediate (especially if it's already publicly known) to a few weeks. For a vulnerability with a straightforward mitigation, we expect report date to disclosure date to be on the order of 7 days. The Kubernetes Security Response Committee holds the final say when setting a disclosure date.


# Dynamic Resource Allocation

Dynamic Resource AllocationFEATURE STATE: Kubernetes v1.34 [stable] (enabled by default: true)This page describes dynamic resource allocation (DRA) in Kubernetes.About DRADRA is a Kubernetes feature that lets you request and share resources among Pods. These resources are often attached devices like hardware accelerators.With DRA, device drivers and cluster admins define device classes that are available to claim in workloads. Kubernetes allocates matching devices to specific claims and places the corresponding Pods on nodes that can access the allocated devices.Allocating resources with DRA is a similar experience to dynamic volume provisioning, in which you use PersistentVolumeClaims to claim storage capacity from storage classes and request the claimed capacity in your Pods.Benefits of DRADRA provides a flexible way to categorize, request, and use devices in your cluster. Using DRA provides benefits like the following:Flexible device filtering: use common expression language (CEL) to perform fine-grained filtering for specific device attributes.Device sharing: share the same resource with multiple containers or Pods by referencing the corresponding resource claim.Centralized device categorization: device drivers and cluster admins can use device classes to provide app operators with hardware categories that are optimized for various use cases. For example, you can create a cost-optimized device class for general-purpose workloads, and a high-performance device class for critical jobs.Simplified Pod requests: with DRA, app operators don't need to specify device quantities in Pod resource requests. Instead, the Pod references a resource claim, and the device configuration in that claim applies to the Pod.These benefits provide significant improvements in the device allocation workflow when compared to device plugins, which require per-container device requests, don't support device sharing, and don't support expression-based device filtering.Types of DRA usersThe workflow of using DRA to allocate devices involves the following types of users:Device owner: responsible for devices. Device owners might be commercial vendors, the cluster operator, or another entity. To use DRA, devices must have DRA-compatible drivers that do the following:Create ResourceSlices that provide Kubernetes with information about nodes and resources.Update ResourceSlices when resource capacity in the cluster changes.Optionally, create DeviceClasses that workload operators can use to claim devices.Cluster admin: responsible for configuring clusters and nodes, attaching devices, installing drivers, and similar tasks. To use DRA, cluster admins do the following:Attach devices to nodes.Install device drivers that support DRA.Optionally, create DeviceClasses that workload operators can use to claim devices.Workload operator: responsible for deploying and managing workloads in the cluster. To use DRA to allocate devices to Pods, workload operators do the following:Create ResourceClaims or ResourceClaimTemplates to request specific configurations within DeviceClasses.Deploy workloads that use specific ResourceClaims or ResourceClaimTemplates.DRA terminologyDRA uses the following Kubernetes API kinds to provide the core allocation functionality. All of these API kinds are included in the resource.k8s.io/v1 API group.DeviceClassDefines a category of devices that can be claimed and how to select specific device attributes in claims. The DeviceClass parameters can match zero or more devices in ResourceSlices. To claim devices from a DeviceClass, ResourceClaims select specific device attributes.ResourceClaimDescribes a request for access to attached resources, such as devices, in the cluster. ResourceClaims provide Pods with access to a specific resource. ResourceClaims can be created by workload operators or generated by Kubernetes based on a ResourceClaimTemplate.ResourceClaimTemplateDefines a template that Kubernetes uses to create per-Pod ResourceClaims for a workload. ResourceClaimTemplates provide Pods with access to separate, similar resources. Each ResourceClaim that Kubernetes generates from the template is bound to a specific Pod. When the Pod terminates, Kubernetes deletes the corresponding ResourceClaim.ResourceSliceRepresents one or more resources that are attached to nodes, such as devices. Drivers create and manage ResourceSlices in the cluster. When a ResourceClaim is created and used in a Pod, Kubernetes uses ResourceSlices to find nodes that have access to the claimed resources. Kubernetes allocates resources to the ResourceClaim and schedules the Pod onto a node that can access the resources.DeviceClassA DeviceClass lets cluster admins or device drivers define categories of devices in the cluster. DeviceClasses tell operators what devices they can request and how they can request those devices. You can use common expression language (CEL) to select devices based on specific attributes. A ResourceClaim that references the DeviceClass can then request specific configurations within the DeviceClass.To create a DeviceClass, see Set Up DRA in a Cluster.ResourceClaims and ResourceClaimTemplatesA ResourceClaim defines the resources that a workload needs. Every ResourceClaim has requests that reference a DeviceClass and select devices from that DeviceClass. ResourceClaims can also use selectors to filter for devices that meet specific requirements, and can use constraints to limit the devices that can satisfy a request. ResourceClaims can be created by workload operators or can be generated by Kubernetes based on a ResourceClaimTemplate. A ResourceClaimTemplate defines a template that Kubernetes can use to auto-generate ResourceClaims for Pods.Use cases for ResourceClaims and ResourceClaimTemplatesThe method that you use depends on your requirements, as follows:ResourceClaim: you want multiple Pods to share access to specific devices. You manually manage the lifecycle of ResourceClaims that you create.ResourceClaimTemplate: you want Pods to have independent access to separate, similarly-configured devices. Kubernetes generates ResourceClaims from the specification in the ResourceClaimTemplate. The lifetime of each generated ResourceClaim is bound to the lifetime of the corresponding Pod.When you define a workload, you can use Common Expression Language (CEL) to filter for specific device attributes or capacity. The available parameters for filtering depend on the device and the drivers.If you directly reference a specific ResourceClaim in a Pod, that ResourceClaim must already exist in the same namespace as the Pod. If the ResourceClaim doesn't exist in the namespace, the Pod won't schedule. This behavior is similar to how a PersistentVolumeClaim must exist in the same namespace as a Pod that references it.You can reference an auto-generated ResourceClaim in a Pod, but this isn't recommended because auto-generated ResourceClaims are bound to the lifetime of the Pod that triggered the generation.To learn how to claim resources using one of these methods, see Allocate Devices to Workloads with DRA.Prioritized listFEATURE STATE: Kubernetes v1.34 [beta] (enabled by default: true)You can provide a prioritized list of subrequests for requests in a ResourceClaim or ResourceClaimTemplate. The scheduler will then select the first subrequest that can be allocated. This allows users to specify alternative devices that can be used by the workload if the primary choice is not available.In the example below, the ResourceClaimTemplate requested a device with the color black and the size large. If a device with those attributes is not available, the pod cannot be scheduled. With the prioritized list feature, a second alternative can be specified, which requests two devices with the color white and size small. The large black device will be allocated if it is available. If it is not, but two small white devices are available, the pod will still be able to run.apiVersion: resource.k8s.io/v1 kind: ResourceClaimTemplate metadata: name: prioritized-list-claim-template spec: spec: devices: requests: - name: req-0 firstAvailable: - name: large-black deviceClassName: resource.example.com selectors: - cel: expression: |- device.attributes["resource-driver.example.com"].color == "black" && device.attributes["resource-driver.example.com"].size == "large" - name: small-white deviceClassName: resource.example.com selectors: - cel: expression: |- device.attributes["resource-driver.example.com"].color == "white" && device.attributes["resource-driver.example.com"].size == "small" count: 2 The decision is made on a per-Pod basis, so if the Pod is a member of a ReplicaSet or similar grouping, you cannot rely on all the members of the group having the same subrequest chosen. Your workload must be able to accommodate this.Prioritized lists is a beta feature and is enabled by default with the DRAPrioritizedList feature gate in the kube-apiserver and kube-scheduler.ResourceSliceEach ResourceSlice represents one or more devices in a pool. The pool is managed by a device driver, which creates and manages ResourceSlices. The resources in a pool might be represented by a single ResourceSlice or span multiple ResourceSlices.ResourceSlices provide useful information to device users and to the scheduler, and are crucial for dynamic resource allocation. Every ResourceSlice must include the following information:Resource pool: a group of one or more resources that the driver manages. The pool can span more than one ResourceSlice. Changes to the resources in a pool must be propagated across all of the ResourceSlices in that pool. The device driver that manages the pool is responsible for ensuring that this propagation happens.Devices: devices in the managed pool. A ResourceSlice can list every device in a pool or a subset of the devices in a pool. The ResourceSlice defines device information like attributes, versions, and capacity. Device users can select devices for allocation by filtering for device information in ResourceClaims or in DeviceClasses.Nodes: the nodes that can access the resources. Drivers can choose which nodes can access the resources, whether that's all of the nodes in the cluster, a single named node, or nodes that have specific node labels.Drivers use a controller to reconcile ResourceSlices in the cluster with the information that the driver has to publish. This controller overwrites any manual changes, such as cluster users creating or modifying ResourceSlices.Consider the following example ResourceSlice:apiVersion: resource.k8s.io/v1 kind: ResourceSlice metadata: name: cat-slice spec: driver: "resource-driver.example.com" pool: generation: 1 name: "black-cat-pool" resourceSliceCount: 1 # The allNodes field defines whether any node in the cluster can access the device. allNodes: true devices: - name: "large-black-cat" attributes: color: string: "black" size: string: "large" cat: boolean: true This ResourceSlice is managed by the resource-driver.example.com driver in the black-cat-pool pool. The allNodes: true field indicates that any node in the cluster can access the devices. There's one device in the ResourceSlice, named large-black-cat, with the following attributes:color: blacksize: largecat: trueA DeviceClass could select this ResourceSlice by using these attributes, and a ResourceClaim could filter for specific devices in that DeviceClass.How resource allocation with DRA worksThe following sections describe the workflow for the various types of DRA users and for the Kubernetes system during dynamic resource allocation.Workflow for usersDriver creation: device owners or third-party entities create drivers that can create and manage ResourceSlices in the cluster. These drivers optionally also create DeviceClasses that define a category of devices and how to request them.Cluster configuration: cluster admins create clusters, attach devices to nodes, and install the DRA device drivers. Cluster admins optionally create DeviceClasses that define categories of devices and how to request them.Resource claims: workload operators create ResourceClaimTemplates or ResourceClaims that request specific device configurations within a DeviceClass. In the same step, workload operators modify their Kubernetes manifests to request those ResourceClaimTemplates or ResourceClaims.Workflow for KubernetesResourceSlice creation: drivers in the cluster create ResourceSlices that represent one or more devices in a managed pool of similar devices.Workload creation: the cluster control plane checks new workloads for references to ResourceClaimTemplates or to specific ResourceClaims.If the workload uses a ResourceClaimTemplate, a controller named the resourceclaim-controller generates ResourceClaims for every Pod in the workload.If the workload uses a specific ResourceClaim, Kubernetes checks whether that ResourceClaim exists in the cluster. If the ResourceClaim doesn't exist, the Pods won't deploy.ResourceSlice filtering: for every Pod, Kubernetes checks the ResourceSlices in the cluster to find a device that satisfies all of the following criteria:The nodes that can access the resources are eligible to run the Pod.The ResourceSlice has unallocated resources that match the requirements of the Pod's ResourceClaim.Resource allocation: after finding an eligible ResourceSlice for a Pod's ResourceClaim, the Kubernetes scheduler updates the ResourceClaim with the allocation details.Pod scheduling: when resource allocation is complete, the scheduler places the Pod on a node that can access the allocated resource. The device driver and the kubelet on that node configure the device and the Pod's access to the device.Observability of dynamic resourcesYou can check the status of dynamically allocated resources by using any of the following methods:kubelet device metricsResourceClaim statusDevice health monitoringkubelet device metricsThe PodResourcesLister kubelet gRPC service lets you monitor in-use devices. The DynamicResource message provides information that's specific to dynamic resource allocation, such as the device name and the claim name. For details, see Monitoring device plugin resources.ResourceClaim device statusFEATURE STATE: Kubernetes v1.33 [beta] (enabled by default: true)DRA drivers can report driver-specific device status data for each allocated device in the status.devices field of a ResourceClaim. For example, the driver might list the IP addresses that are assigned to a network interface device.The accuracy of the information that a driver adds to a ResourceClaim status.devices field depends on the driver. Evaluate drivers to decide whether you can rely on this field as the only source of device information.If you disable the DRAResourceClaimDeviceStatus feature gate, the status.devices field automatically gets cleared when storing the ResourceClaim. A ResourceClaim device status is supported when it is possible, from a DRA driver, to update an existing ResourceClaim where the status.devices field is set.For details about the status.devices field, see the ResourceClaim API reference.Device Health MonitoringFEATURE STATE: Kubernetes v1.31 [alpha] (enabled by default: false)As an alpha feature, Kubernetes provides a mechanism for monitoring and reporting the health of dynamically allocated infrastructure resources. For stateful applications running on specialized hardware, it is critical to know when a device has failed or become unhealthy. It is also helpful to find out if the device recovers.To enable this functionality, the ResourceHealthStatus feature gate must be enabled, and the DRA driver must implement the DRAResourceHealth gRPC service.When a DRA driver detects that an allocated device has become unhealthy, it reports this status back to the kubelet. This health information is then exposed directly in the Pod's status. The kubelet populates the allocatedResourcesStatus field in the status of each container, detailing the health of each device assigned to that container.This provides crucial visibility for users and controllers to react to hardware failures. For a Pod that is failing, you can inspect this status to determine if the failure was related to an unhealthy device.Pre-scheduled PodsWhen you - or another API client - create a Pod with spec.nodeName already set, the scheduler gets bypassed. If some ResourceClaim needed by that Pod does not exist yet, is not allocated or not reserved for the Pod, then the kubelet will fail to run the Pod and re-check periodically because those requirements might still get fulfilled later.Such a situation can also arise when support for dynamic resource allocation was not enabled in the scheduler at the time when the Pod got scheduled (version skew, configuration, feature gate, etc.). kube-controller-manager detects this and tries to make the Pod runnable by reserving the required ResourceClaims. However, this only works if those were allocated by the scheduler for some other pod.It is better to avoid bypassing the scheduler because a Pod that is assigned to a node blocks normal resources (RAM, CPU) that then cannot be used for other Pods while the Pod is stuck. To make a Pod run on a specific node while still going through the normal scheduling flow, create the Pod with a node selector that exactly matches the desired node:apiVersion: v1 kind: Pod metadata: name: pod-with-cats spec: nodeSelector: kubernetes.io/hostname: name-of-the-intended-node ... You may also be able to mutate the incoming Pod, at admission time, to unset the .spec.nodeName field and to use a node selector instead.DRA beta featuresThe following sections describe DRA features that are available in the Beta feature stage. For more information, see Set up DRA in the cluster.Admin accessFEATURE STATE: Kubernetes v1.34 [beta] (enabled by default: true)You can mark a request in a ResourceClaim or ResourceClaimTemplate as having privileged features for maintenance and troubleshooting tasks. A request with admin access grants access to in-use devices and may enable additional permissions when making the device available in a container:apiVersion: resource.k8s.io/v1 kind: ResourceClaimTemplate metadata: name: large-black-cat-claim-template spec: spec: devices: requests: - name: req-0 exactly: deviceClassName: resource.example.com allocationMode: All adminAccess: true If this feature is disabled, the adminAccess field will be removed automatically when creating such a ResourceClaim.Admin access is a privileged mode and should not be granted to regular users in multi-tenant clusters. Starting with Kubernetes v1.33, only users authorized to create ResourceClaim or ResourceClaimTemplate objects in namespaces labeled with resource.k8s.io/admin-access: "true" (case-sensitive) can use the adminAccess field. This ensures that non-admin users cannot misuse the feature. Starting with Kubernetes v1.34, this label has been updated to resource.kubernetes.io/admin-access: "true".DRA alpha featuresThe following sections describe DRA features that are available in the Alpha feature stage. To use any of these features, you must also set up DRA in your clusters by enabling the DynamicResourceAllocation feature gate and the DRA API groups. For more information, see Set up DRA in the cluster.Extended resource allocation by DRAFEATURE STATE: Kubernetes v1.34 [alpha] (enabled by default: false)You can provide an extended resource name for a DeviceClass. The scheduler will then select the devices matching the class for the extended resource requests. This allows users to continue using extended resource requests in a pod to request either extended resources provided by device plugin, or DRA devices. The same extended resource can be provided either by device plugin, or DRA on one single cluster node. The same extended resource can be provided by device plugin on some nodes, and DRA on other nodes in the same cluster.In the example below, the DeviceClass is given an extendedResourceName example.com/gpu. If a pod requested for the extended resource example.com/gpu: 2, it can be scheduled to a node with two or more devices matching the DeviceClass.apiVersion: resource.k8s.io/v1 kind: DeviceClass metadata: name: gpu.example.com spec: selectors: - cel: expression: device.driver == 'gpu.example.com' && device.attributes['gpu.example.com'].type == 'gpu' extendedResourceName: example.com/gpu In addition, users can use a special extended resource to allocate devices without having to explicitly create a ResourceClaim. Using the extended resource name prefix deviceclass.resource.kubernetes.io/ and the DeviceClass name. This works for any DeviceClass, even if it does not specify the an extended resource name. The resulting ResourceClaim will contain a request for an ExactCount of the specified number of devices of that DeviceClass.Extended resource allocation by DRA is an alpha feature and only enabled when the DRAExtendedResource feature gate is enabled in the kube-apiserver, kube-scheduler, and kubelet.Partitionable devicesFEATURE STATE: Kubernetes v1.33 [alpha] (enabled by default: false)Devices represented in DRA don't necessarily have to be a single unit connected to a single machine, but can also be a logical device comprised of multiple devices connected to multiple machines. These devices might consume overlapping resources of the underlying phyical devices, meaning that when one logical device is allocated other devices will no longer be available.In the ResourceSlice API, this is represented as a list of named CounterSets, each of which contains a set of named counters. The counters represent the resources available on the physical device that are used by the logical devices advertised through DRA.Logical devices can specify the ConsumesCounters list. Each entry contains a reference to a CounterSet and a set of named counters with the amounts they will consume. So for a device to be allocatable, the referenced counter sets must have sufficient quantity for the counters referenced by the device.Here is an example of two devices, each consuming 6Gi of memory from the a shared counter with 8Gi of memory. Thus, only one of the devices can be allocated at any point in time. The scheduler handles this and it is transparent to the consumer as the ResourceClaim API is not affected.kind: ResourceSlice apiVersion: resource.k8s.io/v1 metadata: name: resourceslice spec: nodeName: worker-1 pool: name: pool generation: 1 resourceSliceCount: 1 driver: dra.example.com sharedCounters: - name: gpu-1-counters counters: memory: value: 8Gi devices: - name: device-1 consumesCounters: - counterSet: gpu-1-counters counters: memory: value: 6Gi - name: device-2 consumesCounters: - counterSet: gpu-1-counters counters: memory: value: 6Gi Partitionable devices is an alpha feature and only enabled when the DRAPartitionableDevices feature gate is enabled in the kube-apiserver and kube-scheduler.Consumable capacityFEATURE STATE: Kubernetes v1.34 [alpha] (enabled by default: false)The consumable capacity feature allows the same devices to be consumed by multiple independent ResourceClaims, with the Kubernetes scheduler managing how much of the device's capacity is used up by each claim. This is analogous to how Pods can share the resources on a Node; ResourceClaims can share the resources on a Device.The device driver can set allowMultipleAllocations field added in .spec.devices of ResourceSlice to allow allocating that device to multiple independent ResourceClaims or to multiple requests within a ResourceClaim.Users can set capacity field added in spec.devices.requests of ResourceClaim to specify the device resource requirements for each allocation.For the device that allows multiple allocations, the requested capacity is drawn from — or consumed from — its total capacity, a concept known as consumable capacity. Then, the scheduler ensures that the aggregate consumed capacity across all claims does not exceed the device’s overall capacity. Furthermore, driver authors can use the requestPolicy constraints on individual device capacities to control how those capacities are consumed. For example, the driver author can specify that a given capacity is only consumed in increments of 1Gi.Here is an example of a network device which allows multiple allocations and contains a consumable bandwidth capacity.kind: ResourceSlice apiVersion: resource.k8s.io/v1 metadata: name: resourceslice spec: nodeName: worker-1 pool: name: pool generation: 1 resourceSliceCount: 1 driver: dra.example.com devices: - name: eth1 allowMultipleAllocations: true attributes: name: string: "eth1" capacity: bandwidth: requestPolicy: default: "1M" validRange: min: "1M" step: "8" value: "10G" The consumable capacity can be requested as shown in the below example.apiVersion: resource.k8s.io/v1 kind: ResourceClaimTemplate metadata: name: bandwidth-claim-template spec: spec: devices: requests: - name: req-0 exactly: - deviceClassName: resource.example.com capacity: requests: bandwidth: 1G The allocation result will include the consumed capacity and the identifier of the share.apiVersion: resource.k8s.io/v1 kind: ResourceClaim ... status: allocation: devices: results: - consumedCapacity: bandwidth: 1G device: eth1 shareID: "a671734a-e8e5-11e4-8fde-42010af09327" In this example, a multiply-allocatable device was chosen. However, any resource.example.com device with at least the requested 1G bandwidth could have met the requirement. If a non-multiply-allocatable device were chosen, the allocation would have resulted in the entire device. To force the use of a only multiply-allocatable devices, you can use the CEL criteria device.allowMultipleAllocations == true.Device taints and tolerationsFEATURE STATE: Kubernetes v1.33 [alpha] (enabled by default: false)Device taints are similar to node taints: a taint has a string key, a string value, and an effect. The effect is applied to the ResourceClaim which is using a tainted device and to all Pods referencing that ResourceClaim. The "NoSchedule" effect prevents scheduling those Pods. Tainted devices are ignored when trying to allocate a ResourceClaim because using them would prevent scheduling of Pods.The "NoExecute" effect implies "NoSchedule" and in addition causes eviction of all Pods which have been scheduled already. This eviction is implemented in the device taint eviction controller in kube-controller-manager by deleting affected Pods.ResourceClaims can tolerate taints. If a taint is tolerated, its effect does not apply. An empty toleration matches all taints. A toleration can be limited to certain effects and/or match certain key/value pairs. A toleration can check that a certain key exists, regardless which value it has, or it can check for specific values of a key. For more information on this matching see the node taint concepts.Eviction can be delayed by tolerating a taint for a certain duration. That delay starts at the time when a taint gets added to a device, which is recorded in a field of the taint.Taints apply as described above also to ResourceClaims allocating "all" devices on a node. All devices must be untainted or all of their taints must be tolerated. Allocating a device with admin access (described above) is not exempt either. An admin using that mode must explicitly tolerate all taints to access tainted devices.Device taints and tolerations is an alpha feature and only enabled when the DRADeviceTaints feature gate is enabled in the kube-apiserver, kube-controller-manager and kube-scheduler. To use DeviceTaintRules, the resource.k8s.io/v1alpha3 API version must be enabled.You can add taints to devices in the following ways, by using the DeviceTaintRule API kind.Taints set by the driverA DRA driver can add taints to the device information that it publishes in ResourceSlices. Consult the documentation of a DRA driver to learn whether the driver uses taints and what their keys and values are.Taints set by an adminAn admin or a control plane component can taint devices without having to tell the DRA driver to include taints in its device information in ResourceSlices. They do that by creating DeviceTaintRules. Each DeviceTaintRule adds one taint to devices which match the device selector. Without such a selector, no devices are tainted. This makes it harder to accidentally evict all pods using ResourceClaims when leaving out the selector by mistake.Devices can be selected by giving the name of a DeviceClass, driver, pool, and/or device. The DeviceClass selects all devices that are selected by the selectors in that DeviceClass. With just the driver name, an admin can taint all devices managed by that driver, for example while doing some kind of maintenance of that driver across the entire cluster. Adding a pool name can limit the taint to a single node, if the driver manages node-local devices.Finally, adding the device name can select one specific device. The device name and pool name can also be used alone, if desired. For example, drivers for node-local devices are encouraged to use the node name as their pool name. Then tainting with that pool name automatically taints all devices on a node.Drivers might use stable names like "gpu-0" that hide which specific device is currently assigned to that name. To support tainting a specific hardware instance, CEL selectors can be used in a DeviceTaintRule to match a vendor-specific unique ID attribute, if the driver supports one for its hardware.The taint applies as long as the DeviceTaintRule exists. It can be modified and and removed at any time. Here is one example of a DeviceTaintRule for a fictional DRA driver:apiVersion: resource.k8s.io/v1alpha3 kind: DeviceTaintRule metadata: name: example spec: # The entire hardware installation for this # particular driver is broken. # Evict all pods and don't schedule new ones. deviceSelector: driver: dra.example.com taint: key: dra.example.com/unhealthy value: Broken effect: NoExecute Device Binding ConditionsFEATURE STATE: Kubernetes v1.34 [alpha] (enabled by default: false)Device Binding Conditions allow the Kubernetes scheduler to delay Pod binding until external resources, such as fabric-attached GPUs or reprogrammable FPGAs, are confirmed to be ready.This waiting behavior is implemented in the PreBind phase of the scheduling framework. During this phase, the scheduler checks whether all required device conditions are satisfied before proceeding with binding.This improves scheduling reliability by avoiding premature binding and enables coordination with external device controllers.To use this feature, device drivers (typically managed by driver owners) must publish the following fields in the Device section of a ResourceSlice. Cluster administrators must enable the DRADeviceBindingConditions and DRAResourceClaimDeviceStatus feature gates for the scheduler to honor these fields.bindingConditions: A list of condition types that must be set to True in the status.conditions field of the associated ResourceClaim before the Pod can be bound. These typically represent readiness signals such as "DeviceAttached" or "DeviceInitialized".bindingFailureConditions: A list of condition types that, if set to True in status.conditions field of the associated ResourceClaim, indicate a failure state. If any of these conditions are True, the scheduler will abort binding and reschedule the Pod.bindsToNode: if set to true, the scheduler records the selected node name in the status.allocation.nodeSelector field of the ResourceClaim. This does not affect the Pod's spec.nodeSelector. Instead, it sets a node selector inside the ResourceClaim, which external controllers can use to perform node-specific operations such as device attachment or preparation.All condition types listed in bindingConditions and bindingFailureConditions are evaluated from the status.conditions field of the ResourceClaim. External controllers are responsible for updating these conditions using standard Kubernetes condition semantics (type, status, reason, message, lastTransitionTime).The scheduler waits up to 600 seconds for all bindingConditions to become True. If the timeout is reached or any bindingFailureConditions are True, the scheduler clears the allocation and reschedules the Pod.apiVersion: resource.k8s.io/v1 kind: ResourceSlice metadata: name: gpu-slice spec: driver: dra.example.com nodeSelector: accelerator-type: high-performance pool: name: gpu-pool generation: 1 resourceSliceCount: 1 devices: - name: gpu-1 attributes: vendor: string: "example" model: string: "example-gpu" bindsToNode: true bindingConditions: - dra.example.com/is-prepared bindingFailureConditions: - dra.example.com/preparing-failed This example ResourceSlice has the following properties:The ResourceSlice targets nodes labeled with accelerator-type=high-performance, so that the scheduler uses only a specific set of eligible nodes.The scheduler selects one node from the selected group (for example, node-3) and sets the status.allocation.nodeSelector field in the ResourceClaim to that node name.The dra.example.com/is-prepared binding condition indicates that the device gpu-1 must be prepared (the is-prepared condition has a status of True) before binding.If the gpu-1 device preparation fails (the preparing-failed condition has a status of True), the scheduler aborts binding.The scheduler waits up to 600 seconds for the device to become ready.External controllers can use the node selector in the ResourceClaim to perform node-specific setup on the selected node.What's nextSet Up DRA in a ClusterAllocate devices to workloads using DRAFor more information on the design, see the Dynamic Resource Allocation with Structured Parameters KEP.