Install Drivers and Allocate Devices with DRAFEATURE STATE: Kubernetes v1.34 [stable] (enabled by default: true)This tutorial shows you how to install Dynamic Resource Allocation (DRA) drivers in your cluster and how to use them in conjunction with the DRA APIs to allocate devices to Pods. This page is intended for cluster administrators.Dynamic Resource Allocation (DRA) lets a cluster manage availability and allocation of hardware resources to satisfy Pod-based claims for hardware requirements and preferences. To support this, a mixture of Kubernetes built-in components (like the Kubernetes scheduler, kubelet, and kube-controller-manager) and third-party drivers from device owners (called DRA drivers) share the responsibility to advertise, allocate, prepare, mount, healthcheck, unprepare, and cleanup resources throughout the Pod lifecycle. These components share information via a series of DRA specific APIs in the resource.k8s.io API group including DeviceClasses, ResourceSlices, ResourceClaims, as well as new fields in the Pod spec itself.ObjectivesDeploy an example DRA driverDeploy a Pod requesting a hardware claim using DRA APIsDelete a Pod that has a claimBefore you beginYour cluster should support RBAC. You can try this tutorial with a cluster using a different authorization mechanism, but in that case you will have to adapt the steps around defining roles and permissions.You need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. It is recommended to run this tutorial on a cluster with at least two nodes that are not acting as control plane hosts. If you do not already have a cluster, you can create one by using minikube or you can use one of these Kubernetes playgrounds:iximiuz LabsKillercodaKodeKloudPlay with KubernetesThis tutorial has been tested with Linux nodes, though it may also work with other types of nodes.Your Kubernetes server must be version v1.34.To check the version, enter kubectl version.If your cluster is not currently running Kubernetes 1.34 then please check the documentation for the version of Kubernetes that you plan to use.Explore the initial cluster stateYou can spend some time to observe the initial state of a cluster with DRA enabled, especially if you have not used these APIs extensively before. If you set up a new cluster for this tutorial, with no driver installed and no Pod claims yet to satisfy, the output of these commands won't show any resources.Get a list of DeviceClasses:kubectl get deviceclasses The output is similar to this:No resources found Get a list of ResourceSlices:kubectl get resourceslices The output is similar to this:No resources found Get a list of ResourceClaims and ResourceClaimTemplateskubectl get resourceclaims -A kubectl get resourceclaimtemplates -A The output is similar to this:No resources found No resources found At this point, you have confirmed that DRA is enabled and configured properly in the cluster, and that no DRA drivers have advertised any resources to the DRA APIs yet.Install an example DRA driverDRA drivers are third-party applications that run on each node of your cluster to interface with the hardware of that node and Kubernetes' built-in DRA components. The installation procedure depends on the driver you choose, but is likely deployed as a DaemonSet to all or a selection of the nodes (using selectors or similar mechanisms) in your cluster.Check your driver's documentation for specific installation instructions, which might include a Helm chart, a set of manifests, or other deployment tooling.This tutorial uses an example driver which can be found in the kubernetes-sigs/dra-example-driver repository to demonstrate driver installation. This example driver advertises simulated GPUs to Kubernetes for your Pods to interact with.Prepare your cluster for driver installationTo simplify cleanup, create a namespace named dra-tutorial:Create the namespace:kubectl create namespace dra-tutorial In a production environment, you would likely be using a previously released or qualified image from the driver vendor or your own organization, and your nodes would need to have access to the image registry where the driver image is hosted. In this tutorial, you will use a publicly released image of the dra-example-driver to simulate access to a DRA driver image.Confirm your nodes have access to the image by running the following from within one of your cluster's nodes:docker pull registry.k8s.io/dra-example-driver/dra-example-driver:v0.2.0 Deploy the DRA driver componentsFor this tutorial, you will install the critical example resource driver components individually with kubectl.Create the DeviceClass representing the device types this DRA driver supports:dra/driver-install/deviceclass.yaml apiVersion: resource.k8s.io/v1 kind: DeviceClass metadata: name: gpu.example.com spec: selectors: - cel: expression: "device.driver == 'gpu.example.com'"kubectl apply --server-side -f http://k8s.io/examples/dra/driver-install/deviceclass.yaml Create the ServiceAccount, ClusterRole and ClusterRoleBinding that will be used by the driver to gain permissions to interact with the Kubernetes API on this cluster:Create the Service Account:dra/driver-install/serviceaccount.yaml apiVersion: v1 kind: ServiceAccount metadata: name: dra-example-driver-service-account namespace: dra-tutorial labels: app.kubernetes.io/name: dra-example-driver app.kubernetes.io/instance: dra-example-driverkubectl apply --server-side -f http://k8s.io/examples/dra/driver-install/serviceaccount.yaml Create the ClusterRole:dra/driver-install/clusterrole.yaml apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: dra-example-driver-role rules: - apiGroups: ["resource.k8s.io"] resources: ["resourceclaims"] verbs: ["get"] - apiGroups: [""] resources: ["nodes"] verbs: ["get"] - apiGroups: ["resource.k8s.io"] resources: ["resourceslices"] verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]kubectl apply --server-side -f http://k8s.io/examples/dra/driver-install/clusterrole.yaml Create the ClusterRoleBinding:dra/driver-install/clusterrolebinding.yaml apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: dra-example-driver-role-binding subjects: - kind: ServiceAccount name: dra-example-driver-service-account namespace: dra-tutorial roleRef: kind: ClusterRole name: dra-example-driver-role apiGroup: rbac.authorization.k8s.iokubectl apply --server-side -f http://k8s.io/examples/dra/driver-install/clusterrolebinding.yaml Create a PriorityClass for the DRA driver. The PriorityClass prevents preemption of th DRA driver component, which is responsible for important lifecycle operations for Pods with claims. Learn more about pod priority and preemption here.dra/driver-install/priorityclass.yaml apiVersion: scheduling.k8s.io/v1 kind: PriorityClass metadata: name: dra-driver-high-priority value: 1000000 globalDefault: false description: "This priority class should be used for DRA driver pods only."kubectl apply --server-side -f http://k8s.io/examples/dra/driver-install/priorityclass.yaml Deploy the actual DRA driver as a DaemonSet configured to run the example driver binary with the permissions provisioned above. The DaemonSet has the permissions that you granted to the ServiceAccount in the previous steps.dra/driver-install/daemonset.yaml apiVersion: apps/v1 kind: DaemonSet metadata: name: dra-example-driver-kubeletplugin namespace: dra-tutorial labels: app.kubernetes.io/name: dra-example-driver spec: selector: matchLabels: app.kubernetes.io/name: dra-example-driver updateStrategy: type: RollingUpdate template: metadata: labels: app.kubernetes.io/name: dra-example-driver spec: priorityClassName: dra-driver-high-priority serviceAccountName: dra-example-driver-service-account securityContext: {} containers: - name: plugin securityContext: privileged: true image: registry.k8s.io/dra-example-driver/dra-example-driver:v0.2.0 imagePullPolicy: IfNotPresent command: ["dra-example-kubeletplugin"] resources: {} # Production drivers should always implement a liveness probe # For the tutorial we simply omit it # livenessProbe: # grpc: # port: 51515 # service: liveness # failureThreshold: 3 # periodSeconds: 10 env: - name: CDI_ROOT value: /var/run/cdi - name: KUBELET_REGISTRAR_DIRECTORY_PATH value: "/var/lib/kubelet/plugins_registry" - name: KUBELET_PLUGINS_DIRECTORY_PATH value: "/var/lib/kubelet/plugins" - name: NODE_NAME valueFrom: fieldRef: fieldPath: spec.nodeName - name: NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace # Simulated number of devices the example driver will pretend to have. - name: NUM_DEVICES value: "9" - name: HEALTHCHECK_PORT value: "51515" volumeMounts: - name: plugins-registry mountPath: "/var/lib/kubelet/plugins_registry" - name: plugins mountPath: "/var/lib/kubelet/plugins" - name: cdi mountPath: /var/run/cdi volumes: - name: plugins-registry hostPath: path: "/var/lib/kubelet/plugins_registry" - name: plugins hostPath: path: "/var/lib/kubelet/plugins" - name: cdi hostPath: path: /var/run/cdikubectl apply --server-side -f http://k8s.io/examples/dra/driver-install/daemonset.yaml The DaemonSet is configured with the volume mounts necessary to interact with the underlying Container Device Interface (CDI) directory, and to expose its socket to kubelet via the kubelet/plugins directory.Verify the DRA driver installationGet a list of the Pods of the DRA driver DaemonSet across all worker nodes:kubectl get pod -l app.kubernetes.io/name=dra-example-driver -n dra-tutorial The output is similar to this:NAME READY STATUS RESTARTS AGE dra-example-driver-kubeletplugin-4sk2x 1/1 Running 0 13s dra-example-driver-kubeletplugin-cttr2 1/1 Running 0 13s The initial responsibility of each node's local DRA driver is to update the cluster with what devices are available to Pods on that node, by publishing its metadata to the ResourceSlices API. You can check that API to see that each node with a driver is advertising the device class it represents.Check for available ResourceSlices:kubectl get resourceslices The output is similar to this:NAME NODE DRIVER POOL AGE kind-worker-gpu.example.com-k69gd kind-worker gpu.example.com kind-worker 19s kind-worker2-gpu.example.com-qdgpn kind-worker2 gpu.example.com kind-worker2 19s At this point, you have successfully installed the example DRA driver, and confirmed its initial configuration. You're now ready to use DRA to schedule Pods.Claim resources and deploy a PodTo request resources using DRA, you create ResourceClaims or ResourceClaimTemplates that define the resources that your Pods need. In the example driver, a memory capacity attribute is exposed for mock GPU devices. This section shows you how to use Common Expression Language to express your requirements in a ResourceClaim, select that ResourceClaim in a Pod specification, and observe the resource allocation.This tutorial showcases only one basic example of a DRA ResourceClaim. Read Dynamic Resource Allocation to learn more about ResourceClaims.Create the ResourceClaimIn this section, you create a ResourceClaim and reference it in a Pod. Whatever the claim, the deviceClassName is a required field, narrowing down the scope of the request to a specific device class. The request itself can include a Common Expression Language expression that references attributes that may be advertised by the driver managing that device class.In this example, you will create a request for any GPU advertising over 10Gi memory capacity. The attribute exposing capacity from the example driver takes the form device.capacity['gpu.example.com'].memory. Note also that the name of the claim is set to some-gpu.dra/driver-install/example/resourceclaim.yaml apiVersion: resource.k8s.io/v1 kind: ResourceClaim metadata: name: some-gpu namespace: dra-tutorial spec: devices: requests: - name: some-gpu exactly: deviceClassName: gpu.example.com selectors: - cel: expression: "device.capacity['gpu.example.com'].memory.compareTo(quantity('10Gi')) >= 0"kubectl apply --server-side -f http://k8s.io/examples/dra/driver-install/example/resourceclaim.yaml Create the Pod that references that ResourceClaimBelow is the Pod manifest referencing the ResourceClaim you just made, some-gpu, in the spec.resourceClaims.resourceClaimName field. The local name for that claim, gpu, is then used in the spec.containers.resources.claims.name field to allocate the claim to the Pod's underlying container.dra/driver-install/example/pod.yaml apiVersion: v1 kind: Pod metadata: name: pod0 namespace: dra-tutorial labels: app: pod spec: containers: - name: ctr0 image: ubuntu:24.04 command: ["bash", "-c"] args: ["export; trap 'exit 0' TERM; sleep 9999 & wait"] resources: claims: - name: gpu resourceClaims: - name: gpu resourceClaimName: some-gpukubectl apply --server-side -f http://k8s.io/examples/dra/driver-install/example/pod.yaml Confirm the pod has deployed:kubectl get pod pod0 -n dra-tutorial The output is similar to this:NAME READY STATUS RESTARTS AGE pod0 1/1 Running 0 9s Explore the DRA stateAfter you create the Pod, the cluster tries to schedule that Pod to a node where Kubernetes can satisfy the ResourceClaim. In this tutorial, the DRA driver is deployed on all nodes, and is advertising mock GPUs on all nodes, all of which have enough capacity advertised to satisfy the Pod's claim, so Kubernetes can schedule this Pod on any node and can allocate any of the mock GPUs on that node.When Kubernetes allocates a mock GPU to a Pod, the example driver adds environment variables in each container it is allocated to in order to indicate which GPUs would have been injected into them by a real resource driver and how they would have been configured, so you can check those environment variables to see how the Pods have been handled by the system.Check the Pod logs, which report the name of the mock GPU that was allocated:kubectl logs pod0 -c ctr0 -n dra-tutorial | grep -E "GPU_DEVICE_[0-9]+=" | grep -v "RESOURCE_CLAIM" The output is similar to this:declare -x GPU_DEVICE_0="gpu-0" Check the state of the ResourceClaim object:kubectl get resourceclaims -n dra-tutorial The output is similar to this:NAME STATE AGE some-gpu allocated,reserved 34s In this output, the STATE column shows that the ResourceClaim is allocated and reserved.Check the details of the some-gpu ResourceClaim. The status stanza of the ResourceClaim has information about the allocated device and the Pod it has been reserved for:kubectl get resourceclaim some-gpu -n dra-tutorial -o yaml The output is similar to this: 1apiVersion: resource.k8s.io/v1 2kind: ResourceClaim 3metadata: 4 creationTimestamp: "2025-08-20T18:17:31Z" 5 finalizers: 6 - resource.kubernetes.io/delete-protection 7 name: some-gpu 8 namespace: dra-tutorial 9 resourceVersion: "2326" 10 uid: d3e48dbf-40da-47c3-a7b9-f7d54d1051c3 11spec: 12 devices: 13 requests: 14 - exactly: 15 allocationMode: ExactCount 16 count: 1 17 deviceClassName: gpu.example.com 18 selectors: 19 - cel: 20 expression: device.capacity['gpu.example.com'].memory.compareTo(quantity('10Gi')) 21 >= 0 22 name: some-gpu 23status: 24 allocation: 25 devices: 26 results: 27 - device: gpu-0 28 driver: gpu.example.com 29 pool: kind-worker 30 request: some-gpu 31 nodeSelector: 32 nodeSelectorTerms: 33 - matchFields: 34 - key: metadata.name 35 operator: In 36 values: 37 - kind-worker 38 reservedFor: 39 - name: pod0 40 resource: pods 41 uid: c4dadf20-392a-474d-a47b-ab82080c8bd7To check how the driver handled device allocation, get the logs for the driver DaemonSet Pods:kubectl logs -l app.kubernetes.io/name=dra-example-driver -n dra-tutorial The output is similar to this:I0820 18:17:44.131324 1 driver.go:106] PrepareResourceClaims is called: number of claims: 1 I0820 18:17:44.135056 1 driver.go:133] Returning newly prepared devices for claim 'd3e48dbf-40da-47c3-a7b9-f7d54d1051c3': [{[some-gpu] kind-worker gpu-0 [k8s.gpu.example.com/gpu=common k8s.gpu.example.com/gpu=d3e48dbf-40da-47c3-a7b9-f7d54d1051c3-gpu-0]}] You have now successfully deployed a Pod that claims devices using DRA, verified that the Pod was scheduled to an appropriate node, and saw that the associated DRA APIs kinds were updated with the allocation status.Delete a Pod that has a claimWhen a Pod with a claim is deleted, the DRA driver deallocates the resource so it can be available for future scheduling. To validate this behavior, delete the Pod that you created in the previous steps and watch the corresponding changes to the ResourceClaim and driver.Delete the pod0 Pod:kubectl delete pod pod0 -n dra-tutorial The output is similar to this:pod "pod0" deleted Observe the DRA stateWhen the Pod is deleted, the driver deallocates the device from the ResourceClaim and updates the ResourceClaim resource in the Kubernetes API. The ResourceClaim has a pending state until it's referenced in a new Pod.Check the state of the some-gpu ResourceClaim:kubectl get resourceclaims -n dra-tutorial The output is similar to this:NAME STATE AGE some-gpu pending 76s Verify that the driver has processed unpreparing the device for this claim by checking the driver logs:kubectl logs -l app.kubernetes.io/name=dra-example-driver -n dra-tutorial The output is similar to this:I0820 18:22:15.629376 1 driver.go:138] UnprepareResourceClaims is called: number of claims: 1 You have now deleted a Pod that had a claim, and observed that the driver took action to unprepare the underlying hardware resource and update the DRA APIs to reflect that the resource is available again for future scheduling.Cleaning upTo clean up the resources that you created in this tutorial, follow these steps:kubectl delete namespace dra-tutorial kubectl delete deviceclass gpu.example.com kubectl delete clusterrole dra-example-driver-role kubectl delete clusterrolebinding dra-example-driver-role-binding kubectl delete priorityclass dra-driver-high-priority What's nextLearn more about DRAAllocate Devices to Workloads with DRA


Create an External Load BalancerThis page shows how to create an external load balancer.When creating a Service, you have the option of automatically creating a cloud load balancer. This provides an externally-accessible IP address that sends traffic to the correct port on your cluster nodes, provided your cluster runs in a supported environment and is configured with the correct cloud load balancer provider package.You can also use an Ingress in place of Service. For more information, check the Ingress documentation.Before you beginYou need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. It is recommended to run this tutorial on a cluster with at least two nodes that are not acting as control plane hosts. If you do not already have a cluster, you can create one by using minikube or you can use one of these Kubernetes playgrounds:iximiuz LabsKillercodaKodeKloudPlay with KubernetesYour cluster must be running in a cloud or other environment that already has support for configuring external load balancers.Create a ServiceCreate a Service from a manifestTo create an external load balancer, add the following line to your Service manifest: type: LoadBalancer Your manifest might then look like:apiVersion: v1 kind: Service metadata: name: example-service spec: selector: app: example ports: - port: 8765 targetPort: 9376 type: LoadBalancer Create a Service using kubectlYou can alternatively create the service with the kubectl expose command and its --type=LoadBalancer flag:kubectl expose deployment example --port=8765 --target-port=9376 \ --name=example-service --type=LoadBalancer This command creates a new Service using the same selectors as the referenced resource (in the case of the example above, a Deployment named example).For more information, including optional flags, refer to the kubectl expose reference.Finding your IP addressYou can find the IP address created for your service by getting the service information through kubectl:kubectl describe services example-service which should produce output similar to:Name: example-service Namespace: default Labels: app=example Annotations: <none> Selector: app=example Type: LoadBalancer IP Families: <none> IP: 10.3.22.96 IPs: 10.3.22.96 LoadBalancer Ingress: 192.0.2.89 Port: <unset> 8765/TCP TargetPort: 9376/TCP NodePort: <unset> 30593/TCP Endpoints: 172.17.0.3:9376 Session Affinity: None External Traffic Policy: Cluster Events: <none> The load balancer's IP address is listed next to LoadBalancer Ingress.Note:If you are running your service on Minikube, you can find the assigned IP address and port with:minikube service example-service --url Preserving the client source IPBy default, the source IP seen in the target container is not the original source IP of the client. To enable preservation of the client IP, the following fields can be configured in the .spec of the Service:.spec.externalTrafficPolicy - denotes if this Service desires to route external traffic to node-local or cluster-wide endpoints. There are two available options: Cluster (default) and Local. Cluster obscures the client source IP and may cause a second hop to another node, but should have good overall load-spreading. Local preserves the client source IP and avoids a second hop for LoadBalancer and NodePort type Services, but risks potentially imbalanced traffic spreading..spec.healthCheckNodePort - specifies the health check node port (numeric port number) for the service. If you don't specify healthCheckNodePort, the service controller allocates a port from your cluster's NodePort range.You can configure that range by setting an API server command line option, --service-node-port-range. The Service will use the user-specified healthCheckNodePort value if you specify it, provided that the Service type is set to LoadBalancer and externalTrafficPolicy is set to Local.Setting externalTrafficPolicy to Local in the Service manifest activates this feature. For example:apiVersion: v1 kind: Service metadata: name: example-service spec: selector: app: example ports: - port: 8765 targetPort: 9376 externalTrafficPolicy: Local type: LoadBalancer Caveats and limitations when preserving source IPsLoad balancing services from some cloud providers do not let you configure different weights for each target.With each target weighted equally in terms of sending traffic to Nodes, external traffic is not equally load balanced across different Pods. The external load balancer is unaware of the number of Pods on each node that are used as a target.Where NumServicePods << NumNodes or NumServicePods >> NumNodes, a fairly close-to-equal distribution will be seen, even without weights.Internal pod to pod traffic should behave similar to ClusterIP services, with equal probability across all pods.Garbage collecting load balancersFEATURE STATE: Kubernetes v1.17 [stable]In usual case, the correlating load balancer resources in cloud provider should be cleaned up soon after a LoadBalancer type Service is deleted. But it is known that there are various corner cases where cloud resources are orphaned after the associated Service is deleted. Finalizer Protection for Service LoadBalancers was introduced to prevent this from happening. By using finalizers, a Service resource will never be deleted until the correlating load balancer resources are also deleted.Specifically, if a Service has type LoadBalancer, the service controller will attach a finalizer named service.kubernetes.io/load-balancer-cleanup. The finalizer will only be removed after the load balancer resource is cleaned up. This prevents dangling load balancer resources even in corner cases such as the service controller crashing.External load balancer providersIt is important to note that the datapath for this functionality is provided by a load balancer external to the Kubernetes cluster.When the Service type is set to LoadBalancer, Kubernetes provides functionality equivalent to type equals ClusterIP to pods within the cluster and extends it by programming the (external to Kubernetes) load balancer with entries for the nodes hosting the relevant Kubernetes pods. The Kubernetes control plane automates the creation of the external load balancer, health checks (if needed), and packet filtering rules (if needed). Once the cloud provider allocates an IP address for the load balancer, the control plane looks up that external IP address and populates it into the Service object.What's nextFollow the Connecting Applications with Services tutorialRead about ServiceRead about Ingress

Create an External Load Balancer


Example: Deploying Cassandra with a StatefulSetThis tutorial shows you how to run Apache Cassandra on Kubernetes. Cassandra, a database, needs persistent storage to provide data durability (application state). In this example, a custom Cassandra seed provider lets the database discover new Cassandra instances as they join the Cassandra cluster.StatefulSets make it easier to deploy stateful applications into your Kubernetes cluster. For more information on the features used in this tutorial, see StatefulSet.Note:Cassandra and Kubernetes both use the term node to mean a member of a cluster. In this tutorial, the Pods that belong to the StatefulSet are Cassandra nodes and are members of the Cassandra cluster (called a ring). When those Pods run in your Kubernetes cluster, the Kubernetes control plane schedules those Pods onto Kubernetes Nodes.When a Cassandra node starts, it uses a seed list to bootstrap discovery of other nodes in the ring. This tutorial deploys a custom Cassandra seed provider that lets the database discover new Cassandra Pods as they appear inside your Kubernetes cluster.ObjectivesCreate and validate a Cassandra headless Service.Use a StatefulSet to create a Cassandra ring.Validate the StatefulSet.Modify the StatefulSet.Delete the StatefulSet and its Pods.Before you beginYou need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. It is recommended to run this tutorial on a cluster with at least two nodes that are not acting as control plane hosts. If you do not already have a cluster, you can create one by using minikube or you can use one of these Kubernetes playgrounds:iximiuz LabsKillercodaKodeKloudPlay with KubernetesTo complete this tutorial, you should already have a basic familiarity with Pods, Services, and StatefulSets.Additional Minikube setup instructionsCaution:Minikube defaults to 2048MB of memory and 2 CPU. Running Minikube with the default resource configuration results in insufficient resource errors during this tutorial. To avoid these errors, start Minikube with the following settings:minikube start --memory 5120 --cpus=4 Creating a headless Service for CassandraIn Kubernetes, a Service describes a set of Pods that perform the same task.The following Service is used for DNS lookups between Cassandra Pods and clients within your cluster:application/cassandra/cassandra-service.yaml apiVersion: v1 kind: Service metadata: labels: app: cassandra name: cassandra spec: clusterIP: None ports: - port: 9042 selector: app: cassandra Create a Service to track all Cassandra StatefulSet members from the cassandra-service.yaml file:kubectl apply -f https://k8s.io/examples/application/cassandra/cassandra-service.yaml Validating (optional)Get the Cassandra Service.kubectl get svc cassandra The response isNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE cassandra ClusterIP None <none> 9042/TCP 45s If you don't see a Service named cassandra, that means creation failed. Read Debug Services for help troubleshooting common issues.Using a StatefulSet to create a Cassandra ringThe StatefulSet manifest, included below, creates a Cassandra ring that consists of three Pods.Note:This example uses the default provisioner for Minikube. Please update the following StatefulSet for the cloud you are working with.application/cassandra/cassandra-statefulset.yaml apiVersion: apps/v1 kind: StatefulSet metadata: name: cassandra labels: app: cassandra spec: serviceName: cassandra replicas: 3 selector: matchLabels: app: cassandra template: metadata: labels: app: cassandra spec: terminationGracePeriodSeconds: 500 containers: - name: cassandra image: gcr.io/google-samples/cassandra:v13 imagePullPolicy: Always ports: - containerPort: 7000 name: intra-node - containerPort: 7001 name: tls-intra-node - containerPort: 7199 name: jmx - containerPort: 9042 name: cql resources: limits: cpu: "500m" memory: 1Gi requests: cpu: "500m" memory: 1Gi securityContext: capabilities: add: - IPC_LOCK lifecycle: preStop: exec: command: - /bin/sh - -c - nodetool drain env: - name: MAX_HEAP_SIZE value: 512M - name: HEAP_NEWSIZE value: 100M - name: CASSANDRA_SEEDS value: "cassandra-0.cassandra.default.svc.cluster.local" - name: CASSANDRA_CLUSTER_NAME value: "K8Demo" - name: CASSANDRA_DC value: "DC1-K8Demo" - name: CASSANDRA_RACK value: "Rack1-K8Demo" - name: POD_IP valueFrom: fieldRef: fieldPath: status.podIP readinessProbe: exec: command: - /bin/bash - -c - /ready-probe.sh initialDelaySeconds: 15 timeoutSeconds: 5 # These volume mounts are persistent. They are like inline claims, # but not exactly because the names need to match exactly one of # the stateful pod volumes. volumeMounts: - name: cassandra-data mountPath: /cassandra_data # These are converted to volume claims by the controller # and mounted at the paths mentioned above. # do not use these in production until ssd GCEPersistentDisk or other ssd pd volumeClaimTemplates: - metadata: name: cassandra-data spec: accessModes: [ "ReadWriteOnce" ] storageClassName: fast resources: requests: storage: 1Gi --- kind: StorageClass apiVersion: storage.k8s.io/v1 metadata: name: fast provisioner: k8s.io/minikube-hostpath parameters: type: pd-ssd Create the Cassandra StatefulSet from the cassandra-statefulset.yaml file:# Use this if you are able to apply cassandra-statefulset.yaml unmodified kubectl apply -f https://k8s.io/examples/application/cassandra/cassandra-statefulset.yaml If you need to modify cassandra-statefulset.yaml to suit your cluster, download https://k8s.io/examples/application/cassandra/cassandra-statefulset.yaml and then apply that manifest, from the folder you saved the modified version into:# Use this if you needed to modify cassandra-statefulset.yaml locally kubectl apply -f cassandra-statefulset.yaml Validating the Cassandra StatefulSetGet the Cassandra StatefulSet:kubectl get statefulset cassandra The response should be similar to:NAME DESIRED CURRENT AGE cassandra 3 0 13s The StatefulSet resource deploys Pods sequentially.Get the Pods to see the ordered creation status:kubectl get pods -l="app=cassandra" The response should be similar to:NAME READY STATUS RESTARTS AGE cassandra-0 1/1 Running 0 1m cassandra-1 0/1 ContainerCreating 0 8s It can take several minutes for all three Pods to deploy. Once they are deployed, the same command returns output similar to:NAME READY STATUS RESTARTS AGE cassandra-0 1/1 Running 0 10m cassandra-1 1/1 Running 0 9m cassandra-2 1/1 Running 0 8m Run the Cassandra nodetool inside the first Pod, to display the status of the ring.kubectl exec -it cassandra-0 -- nodetool status The response should look something like:Datacenter: DC1-K8Demo ====================== Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns (effective) Host ID Rack UN 172.17.0.5 83.57 KiB 32 74.0% e2dd09e6-d9d3-477e-96c5-45094c08db0f Rack1-K8Demo UN 172.17.0.4 101.04 KiB 32 58.8% f89d6835-3a42-4419-92b3-0e62cae1479c Rack1-K8Demo UN 172.17.0.6 84.74 KiB 32 67.1% a6a1e8c2-3dc5-4417-b1a0-26507af2aaad Rack1-K8Demo Modifying the Cassandra StatefulSetUse kubectl edit to modify the size of a Cassandra StatefulSet.Run the following command:kubectl edit statefulset cassandra This command opens an editor in your terminal. The line you need to change is the replicas field. The following sample is an excerpt of the StatefulSet file:# Please edit the object below. Lines beginning with a '#' will be ignored, # and an empty file will abort the edit. If an error occurs while saving this file will be # reopened with the relevant failures. # apiVersion: apps/v1 kind: StatefulSet metadata: creationTimestamp: 2016-08-13T18:40:58Z generation: 1 labels: app: cassandra name: cassandra namespace: default resourceVersion: "323" uid: 7a219483-6185-11e6-a910-42010a8a0fc0 spec: replicas: 3 Change the number of replicas to 4, and then save the manifest.The StatefulSet now scales to run with 4 Pods.Get the Cassandra StatefulSet to verify your change:kubectl get statefulset cassandra The response should be similar to:NAME DESIRED CURRENT AGE cassandra 4 4 36m Cleaning upDeleting or scaling a StatefulSet down does not delete the volumes associated with the StatefulSet. This setting is for your safety because your data is more valuable than automatically purging all related StatefulSet resources.Warning:Depending on the storage class and reclaim policy, deleting the PersistentVolumeClaims may cause the associated volumes to also be deleted. Never assume you'll be able to access data if its volume claims are deleted.Run the following commands (chained together into a single command) to delete everything in the Cassandra StatefulSet:grace=$(kubectl get pod cassandra-0 -o=jsonpath='{.spec.terminationGracePeriodSeconds}') \ && kubectl delete statefulset -l app=cassandra \ && echo "Sleeping ${grace} seconds" 1>&2 \ && sleep $grace \ && kubectl delete persistentvolumeclaim -l app=cassandra Run the following command to delete the Service you set up for Cassandra:kubectl delete service -l app=cassandra Cassandra container environment variablesThe Pods in this tutorial use the gcr.io/google-samples/cassandra:v13 image from Google's container registry. The Docker image above is based on debian-base and includes OpenJDK 8.This image includes a standard Cassandra installation from the Apache Debian repo. By using environment variables you can change values that are inserted into cassandra.yaml.Environment variableDefault valueCASSANDRA_CLUSTER_NAME'Test Cluster'CASSANDRA_NUM_TOKENS32CASSANDRA_RPC_ADDRESS0.0.0.0What's nextLearn how to Scale a StatefulSet.Learn more about the KubernetesSeedProviderSee more custom Seed Provider Configurations


CSINodeCSINode holds information about all CSI drivers installed on a node.apiVersion: storage.k8s.io/v1import "k8s.io/api/storage/v1"CSINodeCSINode holds information about all CSI drivers installed on a node. CSI drivers do not need to create the CSINode object directly. As long as they use the node-driver-registrar sidecar container, the kubelet will automatically populate the CSINode object for the CSI driver as part of kubelet plugin registration. CSINode has the same name as a node. If the object is missing, it means either there are no CSI Drivers available on the node, or the Kubelet version is low enough that it doesn't create this object. CSINode has an OwnerReference that points to the corresponding node object.apiVersion: storage.k8s.io/v1kind: CSINodemetadata (ObjectMeta)Standard object's metadata. metadata.name must be the Kubernetes node name.spec (CSINodeSpec), requiredspec is the specification of CSINodeCSINodeSpecCSINodeSpec holds information about the specification of all CSI drivers installed on a nodedrivers ([]CSINodeDriver), requiredPatch strategy: merge on key nameMap: unique values on key name will be kept during a mergedrivers is a list of information of all CSI Drivers existing on a node. If all drivers in the list are uninstalled, this can become empty.CSINodeDriver holds information about the specification of one CSI driver installed on a nodedrivers.name (string), requiredname represents the name of the CSI driver that this object refers to. This MUST be the same name returned by the CSI GetPluginName() call for that driver.drivers.nodeID (string), requirednodeID of the node from the driver point of view. This field enables Kubernetes to communicate with storage systems that do not share the same nomenclature for nodes. For example, Kubernetes may refer to a given node as "node1", but the storage system may refer to the same node as "nodeA". When Kubernetes issues a command to the storage system to attach a volume to a specific node, it can use this field to refer to the node name using the ID that the storage system will understand, e.g. "nodeA" instead of "node1". This field is required.drivers.allocatable (VolumeNodeResources)allocatable represents the volume resources of a node that are available for scheduling. This field is beta.VolumeNodeResources is a set of resource limits for scheduling of volumes.drivers.allocatable.count (int32)count indicates the maximum number of unique volumes managed by the CSI driver that can be used on a node. A volume that is both attached and mounted on a node is considered to be used once, not twice. The same rule applies for a unique volume that is shared among multiple pods on the same node. If this field is not specified, then the supported number of volumes on this node is unbounded.drivers.topologyKeys ([]string)Atomic: will be replaced during a mergetopologyKeys is the list of keys supported by the driver. When a driver is initialized on a cluster, it provides a set of topology keys that it understands (e.g. "company.com/zone", "company.com/region"). When a driver is initialized on a node, it provides the same topology keys along with values. Kubelet will expose these topology keys as labels on its own node object. When Kubernetes does topology aware provisioning, it can use this list to determine which labels it should retrieve from the node object and pass back to the driver. It is possible for different nodes to use different topology keys. This can be empty if driver does not support topology.CSINodeListCSINodeList is a collection of CSINode objects.apiVersion: storage.k8s.io/v1kind: CSINodeListmetadata (ListMeta)Standard list metadata More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadataitems ([]CSINode), requireditems is the list of CSINodeOperationsget read the specified CSINodeHTTP RequestGET /apis/storage.k8s.io/v1/csinodes/{name}Parametersname (in path): string, requiredname of the CSINodepretty (in query): stringprettyResponse200 (CSINode): OK401: Unauthorizedlist list or watch objects of kind CSINodeHTTP RequestGET /apis/storage.k8s.io/v1/csinodesParametersallowWatchBookmarks (in query): booleanallowWatchBookmarkscontinue (in query): stringcontinuefieldSelector (in query): stringfieldSelectorlabelSelector (in query): stringlabelSelectorlimit (in query): integerlimitpretty (in query): stringprettyresourceVersion (in query): stringresourceVersionresourceVersionMatch (in query): stringresourceVersionMatchsendInitialEvents (in query): booleansendInitialEventstimeoutSeconds (in query): integertimeoutSecondswatch (in query): booleanwatchResponse200 (CSINodeList): OK401: Unauthorizedcreate create a CSINodeHTTP RequestPOST /apis/storage.k8s.io/v1/csinodesParametersbody: CSINode, requireddryRun (in query): stringdryRunfieldManager (in query): stringfieldManagerfieldValidation (in query): stringfieldValidationpretty (in query): stringprettyResponse200 (CSINode): OK201 (CSINode): Created202 (CSINode): Accepted401: Unauthorizedupdate replace the specified CSINodeHTTP RequestPUT /apis/storage.k8s.io/v1/csinodes/{name}Parametersname (in path): string, requiredname of the CSINodebody: CSINode, requireddryRun (in query): stringdryRunfieldManager (in query): stringfieldManagerfieldValidation (in query): stringfieldValidationpretty (in query): stringprettyResponse200 (CSINode): OK201 (CSINode): Created401: Unauthorizedpatch partially update the specified CSINodeHTTP RequestPATCH /apis/storage.k8s.io/v1/csinodes/{name}Parametersname (in path): string, requiredname of the CSINodebody: Patch, requireddryRun (in query): stringdryRunfieldManager (in query): stringfieldManagerfieldValidation (in query): stringfieldValidationforce (in query): booleanforcepretty (in query): stringprettyResponse200 (CSINode): OK201 (CSINode): Created401: Unauthorizeddelete delete a CSINodeHTTP RequestDELETE /apis/storage.k8s.io/v1/csinodes/{name}Parametersname (in path): string, requiredname of the CSINodebody: DeleteOptionsdryRun (in query): stringdryRungracePeriodSeconds (in query): integergracePeriodSecondsignoreStoreReadErrorWithClusterBreakingPotential (in query): booleanignoreStoreReadErrorWithClusterBreakingPotentialpretty (in query): stringprettypropagationPolicy (in query): stringpropagationPolicyResponse200 (CSINode): OK202 (CSINode): Accepted401: Unauthorizeddeletecollection delete collection of CSINodeHTTP RequestDELETE /apis/storage.k8s.io/v1/csinodesParametersbody: DeleteOptionscontinue (in query): stringcontinuedryRun (in query): stringdryRunfieldSelector (in query): stringfieldSelectorgracePeriodSeconds (in query): integergracePeriodSecondsignoreStoreReadErrorWithClusterBreakingPotential (in query): booleanignoreStoreReadErrorWithClusterBreakingPotentiallabelSelector (in query): stringlabelSelectorlimit (in query): integerlimitpretty (in query): stringprettypropagationPolicy (in query): stringpropagationPolicyresourceVersion (in query): stringresourceVersionresourceVersionMatch (in query): stringresourceVersionMatchsendInitialEvents (in query): booleansendInitialEventstimeoutSeconds (in query): integertimeoutSecondsResponse200 (Status): OK401: UnauthorizedThis page is automatically generated.If you plan to report an issue with this page, mention that the page is auto-generated in your issue description. The fix may need to happen elsewhere in the Kubernetes project.

CSINode


Operator patternOperators are software extensions to Kubernetes that make use of custom resources to manage applications and their components. Operators follow Kubernetes principles, notably the control loop.MotivationThe operator pattern aims to capture the key aim of a human operator who is managing a service or set of services. Human operators who look after specific applications and services have deep knowledge of how the system ought to behave, how to deploy it, and how to react if there are problems.People who run workloads on Kubernetes often like to use automation to take care of repeatable tasks. The operator pattern captures how you can write code to automate a task beyond what Kubernetes itself provides.Operators in KubernetesKubernetes is designed for automation. Out of the box, you get lots of built-in automation from the core of Kubernetes. You can use Kubernetes to automate deploying and running workloads, and you can automate how Kubernetes does that.Kubernetes' operator pattern concept lets you extend the cluster's behaviour without modifying the code of Kubernetes itself by linking controllers to one or more custom resources. Operators are clients of the Kubernetes API that act as controllers for a Custom Resource.An example operatorSome of the things that you can use an operator to automate include:deploying an application on demandtaking and restoring backups of that application's statehandling upgrades of the application code alongside related changes such as database schemas or extra configuration settingspublishing a Service to applications that don't support Kubernetes APIs to discover themsimulating failure in all or part of your cluster to test its resiliencechoosing a leader for a distributed application without an internal member election processWhat might an operator look like in more detail? Here's an example:A custom resource named SampleDB, that you can configure into the cluster.A Deployment that makes sure a Pod is running that contains the controller part of the operator.A container image of the operator code.Controller code that queries the control plane to find out what SampleDB resources are configured.The core of the operator is code to tell the API server how to make reality match the configured resources.If you add a new SampleDB, the operator sets up PersistentVolumeClaims to provide durable database storage, a StatefulSet to run SampleDB and a Job to handle initial configuration.If you delete it, the operator takes a snapshot, then makes sure that the StatefulSet and Volumes are also removed.The operator also manages regular database backups. For each SampleDB resource, the operator determines when to create a Pod that can connect to the database and take backups. These Pods would rely on a ConfigMap and / or a Secret that has database connection details and credentials.Because the operator aims to provide robust automation for the resource it manages, there would be additional supporting code. For this example, code checks to see if the database is running an old version and, if so, creates Job objects that upgrade it for you.Deploying operatorsThe most common way to deploy an operator is to add the Custom Resource Definition and its associated Controller to your cluster. The Controller will normally run outside of the control plane, much as you would run any containerized application. For example, you can run the controller in your cluster as a Deployment.Using an operatorOnce you have an operator deployed, you'd use it by adding, modifying or deleting the kind of resource that the operator uses. Following the above example, you would set up a Deployment for the operator itself, and then:kubectl get SampleDB # find configured databases kubectl edit SampleDB/example-database # manually change some settings …and that's it! The operator will take care of applying the changes as well as keeping the existing service in good shape.Writing your own operatorIf there isn't an operator in the ecosystem that implements the behavior you want, you can code your own.You also implement an operator (that is, a Controller) using any language / runtime that can act as a client for the Kubernetes API.Following are a few libraries and tools you can use to write your own cloud native operator.Note: This section links to third party projects that provide functionality required by Kubernetes. The Kubernetes project authors aren't responsible for these projects, which are listed alphabetically. To add a project to this list, read the content guide before submitting a change. More information.Charmed Operator FrameworkJava Operator SDKKopf (Kubernetes Operator Pythonic Framework)kube-rs (Rust)kubebuilderKubeOps (.NET operator SDK)MastMetacontroller along with WebHooks that you implement yourselfOperator Frameworkshell-operatorWhat's nextRead the CNCF Operator White Paper.Learn more about Custom ResourcesFind ready-made operators on OperatorHub.io to suit your use casePublish your operator for other people to useRead CoreOS' original article that introduced the operator pattern (this is an archived version of the original article).Read an article from Google Cloud about best practices for building operatorsItems on this page refer to third party products or projects that provide functionality required by Kubernetes. The Kubernetes project authors aren't responsible for those third-party products or projects. See the CNCF website guidelines for more details.You should read the content guide before proposing a change that adds an extra third-party link.

Operator pattern


Configure a Security Context for a Pod or ContainerA security context defines privilege and access control settings for a Pod or Container. Security context settings include, but are not limited to:Discretionary Access Control: Permission to access an object, like a file, is based on user ID (UID) and group ID (GID).Security Enhanced Linux (SELinux): Objects are assigned security labels.Running as privileged or unprivileged.Linux Capabilities: Give a process some privileges, but not all the privileges of the root user.AppArmor: Use program profiles to restrict the capabilities of individual programs.Seccomp: Filter a process's system calls.allowPrivilegeEscalation: Controls whether a process can gain more privileges than its parent process. This bool directly controls whether the no_new_privs flag gets set on the container process. allowPrivilegeEscalation is always true when the container:is run as privileged, orhas CAP_SYS_ADMINreadOnlyRootFilesystem: Mounts the container's root filesystem as read-only.The above bullets are not a complete set of security context settings -- please see SecurityContext for a comprehensive list.Before you beginYou need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. It is recommended to run this tutorial on a cluster with at least two nodes that are not acting as control plane hosts. If you do not already have a cluster, you can create one by using minikube or you can use one of these Kubernetes playgrounds:iximiuz LabsKillercodaKodeKloudPlay with KubernetesTo check the version, enter kubectl version.Set the security context for a PodTo specify security settings for a Pod, include the securityContext field in the Pod specification. The securityContext field is a PodSecurityContext object. The security settings that you specify for a Pod apply to all Containers in the Pod. Here is a configuration file for a Pod that has a securityContext and an emptyDir volume:pods/security/security-context.yaml apiVersion: v1 kind: Pod metadata: name: security-context-demo spec: securityContext: runAsUser: 1000 runAsGroup: 3000 fsGroup: 2000 supplementalGroups: [4000] volumes: - name: sec-ctx-vol emptyDir: {} containers: - name: sec-ctx-demo image: busybox:1.28 command: [ "sh", "-c", "sleep 1h" ] volumeMounts: - name: sec-ctx-vol mountPath: /data/demo securityContext: allowPrivilegeEscalation: false In the configuration file, the runAsUser field specifies that for any Containers in the Pod, all processes run with user ID 1000. The runAsGroup field specifies the primary group ID of 3000 for all processes within any containers of the Pod. If this field is omitted, the primary group ID of the containers will be root(0). Any files created will also be owned by user 1000 and group 3000 when runAsGroup is specified. Since fsGroup field is specified, all processes of the container are also part of the supplementary group ID 2000. The owner for volume /data/demo and any files created in that volume will be Group ID 2000. Additionally, when the supplementalGroups field is specified, all processes of the container are also part of the specified groups. If this field is omitted, it means empty.Create the Pod:kubectl apply -f https://k8s.io/examples/pods/security/security-context.yaml Verify that the Pod's Container is running:kubectl get pod security-context-demo Get a shell to the running Container:kubectl exec -it security-context-demo -- sh In your shell, list the running processes:ps The output shows that the processes are running as user 1000, which is the value of runAsUser:PID USER TIME COMMAND 1 1000 0:00 sleep 1h 6 1000 0:00 sh ... In your shell, navigate to /data, and list the one directory:cd /data ls -l The output shows that the /data/demo directory has group ID 2000, which is the value of fsGroup.drwxrwsrwx 2 root 2000 4096 Jun 6 20:08 demo In your shell, navigate to /data/demo, and create a file:cd demo echo hello > testfile List the file in the /data/demo directory:ls -l The output shows that testfile has group ID 2000, which is the value of fsGroup.-rw-r--r-- 1 1000 2000 6 Jun 6 20:08 testfile Run the following command:id The output is similar to this:uid=1000 gid=3000 groups=2000,3000,4000 From the output, you can see that gid is 3000 which is same as the runAsGroup field. If the runAsGroup was omitted, the gid would remain as 0 (root) and the process will be able to interact with files that are owned by the root(0) group and groups that have the required group permissions for the root (0) group. You can also see that groups contains the group IDs which are specified by fsGroup and supplementalGroups, in addition to gid.Exit your shell:exit Implicit group memberships defined in /etc/group in the container imageBy default, kubernetes merges group information from the Pod with information defined in /etc/group in the container image.pods/security/security-context-5.yaml apiVersion: v1 kind: Pod metadata: name: security-context-demo spec: securityContext: runAsUser: 1000 runAsGroup: 3000 supplementalGroups: [4000] containers: - name: sec-ctx-demo image: registry.k8s.io/e2e-test-images/agnhost:2.45 command: [ "sh", "-c", "sleep 1h" ] securityContext: allowPrivilegeEscalation: false This Pod security context contains runAsUser, runAsGroup and supplementalGroups. However, you can see that the actual supplementary groups attached to the container process will include group IDs which come from /etc/group in the container image.Create the Pod:kubectl apply -f https://k8s.io/examples/pods/security/security-context-5.yaml Verify that the Pod's Container is running:kubectl get pod security-context-demo Get a shell to the running Container:kubectl exec -it security-context-demo -- sh Check the process identity:$ id The output is similar to this:uid=1000 gid=3000 groups=3000,4000,50000 You can see that groups includes group ID 50000. This is because the user (uid=1000), which is defined in the image, belongs to the group (gid=50000), which is defined in /etc/group inside the container image.Check the /etc/group in the container image:$ cat /etc/group You can see that uid 1000 belongs to group 50000.... user-defined-in-image:x:1000: group-defined-in-image:x:50000:user-defined-in-image Exit your shell:exit Note:Implicitly merged supplementary groups may cause security problems particularly when accessing the volumes (see kubernetes/kubernetes#112879 for details). If you want to avoid this. Please see the below section.Configure fine-grained SupplementalGroups control for a PodFEATURE STATE: Kubernetes v1.33 [beta] (enabled by default: true)This feature can be enabled by setting the SupplementalGroupsPolicy feature gate for kubelet and kube-apiserver, and setting the .spec.securityContext.supplementalGroupsPolicy field for a pod.The supplementalGroupsPolicy field defines the policy for calculating the supplementary groups for the container processes in a pod. There are two valid values for this field:Merge: The group membership defined in /etc/group for the container's primary user will be merged. This is the default policy if not specified.Strict: Only group IDs in fsGroup, supplementalGroups, or runAsGroup fields are attached as the supplementary groups of the container processes. This means no group membership from /etc/group for the container's primary user will be merged.When the feature is enabled, it also exposes the process identity attached to the first container process in .status.containerStatuses[].user.linux field. It would be useful for detecting if implicit group ID's are attached.pods/security/security-context-6.yaml apiVersion: v1 kind: Pod metadata: name: security-context-demo spec: securityContext: runAsUser: 1000 runAsGroup: 3000 supplementalGroups: [4000] supplementalGroupsPolicy: Strict containers: - name: sec-ctx-demo image: registry.k8s.io/e2e-test-images/agnhost:2.45 command: [ "sh", "-c", "sleep 1h" ] securityContext: allowPrivilegeEscalation: false This pod manifest defines supplementalGroupsPolicy=Strict. You can see that no group memberships defined in /etc/group are merged to the supplementary groups for container processes.Create the Pod:kubectl apply -f https://k8s.io/examples/pods/security/security-context-6.yaml Verify that the Pod's Container is running:kubectl get pod security-context-demo Check the process identity:kubectl exec -it security-context-demo -- id The output is similar to this:uid=1000 gid=3000 groups=3000,4000 See the Pod's status:kubectl get pod security-context-demo -o yaml You can see that the status.containerStatuses[].user.linux field exposes the process identity attached to the first container process.... status: containerStatuses: - name: sec-ctx-demo user: linux: gid: 3000 supplementalGroups: - 3000 - 4000 uid: 1000 ... Note:Please note that the values in the status.containerStatuses[].user.linux field is the first attached process identity to the first container process in the container. If the container has sufficient privilege to make system calls related to process identity (e.g. setuid(2), setgid(2) or setgroups(2), etc.), the container process can change its identity. Thus, the actual process identity will be dynamic.ImplementationsNote: This section links to third party projects that provide functionality required by Kubernetes. The Kubernetes project authors aren't responsible for these projects, which are listed alphabetically. To add a project to this list, read the content guide before submitting a change. More information.The following container runtimes are known to support fine-grained SupplementalGroups control.CRI-level:containerd, since v2.0CRI-O, since v1.31You can see if the feature is supported in the Node status.apiVersion: v1 kind: Node ... status: features: supplementalGroupsPolicy: true Note:At this alpha release(from v1.31 to v1.32), when a pod with SupplementalGroupsPolicy=Strict are scheduled to a node that does NOT support this feature(i.e. .status.features.supplementalGroupsPolicy=false), the pod's supplemental groups policy falls back to the Merge policy silently.However, since the beta release (v1.33), to enforce the policy more strictly, such pod creation will be rejected by kubelet because the node cannot ensure the specified policy. When your pod is rejected, you will see warning events with reason=SupplementalGroupsPolicyNotSupported like below:apiVersion: v1 kind: Event ... type: Warning reason: SupplementalGroupsPolicyNotSupported message: "SupplementalGroupsPolicy=Strict is not supported in this node" involvedObject: apiVersion: v1 kind: Pod ... Configure volume permission and ownership change policy for PodsFEATURE STATE: Kubernetes v1.23 [stable]By default, Kubernetes recursively changes ownership and permissions for the contents of each volume to match the fsGroup specified in a Pod's securityContext when that volume is mounted. For large volumes, checking and changing ownership and permissions can take a lot of time, slowing Pod startup. You can use the fsGroupChangePolicy field inside a securityContext to control the way that Kubernetes checks and manages ownership and permissions for a volume.fsGroupChangePolicy - fsGroupChangePolicy defines behavior for changing ownership and permission of the volume before being exposed inside a Pod. This field only applies to volume types that support fsGroup controlled ownership and permissions. This field has two possible values:OnRootMismatch: Only change permissions and ownership if the permission and the ownership of root directory does not match with expected permissions of the volume. This could help shorten the time it takes to change ownership and permission of a volume.Always: Always change permission and ownership of the volume when volume is mounted.For example:securityContext: runAsUser: 1000 runAsGroup: 3000 fsGroup: 2000 fsGroupChangePolicy: "OnRootMismatch" Note:This field has no effect on ephemeral volume types such as secret, configMap, and emptyDir.Delegating volume permission and ownership change to CSI driverFEATURE STATE: Kubernetes v1.26 [stable]If you deploy a Container Storage Interface (CSI) driver which supports the VOLUME_MOUNT_GROUP NodeServiceCapability, the process of setting file ownership and permissions based on the fsGroup specified in the securityContext will be performed by the CSI driver instead of Kubernetes. In this case, since Kubernetes doesn't perform any ownership and permission change, fsGroupChangePolicy does not take effect, and as specified by CSI, the driver is expected to mount the volume with the provided fsGroup, resulting in a volume that is readable/writable by the fsGroup.Set the security context for a ContainerTo specify security settings for a Container, include the securityContext field in the Container manifest. The securityContext field is a SecurityContext object. Security settings that you specify for a Container apply only to the individual Container, and they override settings made at the Pod level when there is overlap. Container settings do not affect the Pod's Volumes.Here is the configuration file for a Pod that has one Container. Both the Pod and the Container have a securityContext field:pods/security/security-context-2.yaml apiVersion: v1 kind: Pod metadata: name: security-context-demo-2 spec: securityContext: runAsUser: 1000 containers: - name: sec-ctx-demo-2 image: gcr.io/google-samples/hello-app:2.0 securityContext: runAsUser: 2000 allowPrivilegeEscalation: false Create the Pod:kubectl apply -f https://k8s.io/examples/pods/security/security-context-2.yaml Verify that the Pod's Container is running:kubectl get pod security-context-demo-2 Get a shell into the running Container:kubectl exec -it security-context-demo-2 -- sh In your shell, list the running processes:ps aux The output shows that the processes are running as user 2000. This is the value of runAsUser specified for the Container. It overrides the value 1000 that is specified for the Pod.USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND 2000 1 0.0 0.0 4336 764 ? Ss 20:36 0:00 /bin/sh -c node server.js 2000 8 0.1 0.5 772124 22604 ? Sl 20:36 0:00 node server.js ... Exit your shell:exit Set capabilities for a ContainerWith Linux capabilities, you can grant certain privileges to a process without granting all the privileges of the root user. To add or remove Linux capabilities for a Container, include the capabilities field in the securityContext section of the Container manifest.First, see what happens when you don't include a capabilities field. Here is configuration file that does not add or remove any Container capabilities:pods/security/security-context-3.yaml apiVersion: v1 kind: Pod metadata: name: security-context-demo-3 spec: containers: - name: sec-ctx-3 image: gcr.io/google-samples/hello-app:2.0 Create the Pod:kubectl apply -f https://k8s.io/examples/pods/security/security-context-3.yaml Verify that the Pod's Container is running:kubectl get pod security-context-demo-3 Get a shell into the running Container:kubectl exec -it security-context-demo-3 -- sh In your shell, list the running processes:ps aux The output shows the process IDs (PIDs) for the Container:USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 1 0.0 0.0 4336 796 ? Ss 18:17 0:00 /bin/sh -c node server.js root 5 0.1 0.5 772124 22700 ? Sl 18:17 0:00 node server.js In your shell, view the status for process 1:cd /proc/1 cat status The output shows the capabilities bitmap for the process:... CapPrm: 00000000a80425fb CapEff: 00000000a80425fb ... Make a note of the capabilities bitmap, and then exit your shell:exit


Extend Service IP RangesFEATURE STATE: Kubernetes v1.33 [stable] (enabled by default: true)This document shares how to extend the existing Service IP range assigned to a cluster.Before you beginYou need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. It is recommended to run this tutorial on a cluster with at least two nodes that are not acting as control plane hosts. If you do not already have a cluster, you can create one by using minikube or you can use one of these Kubernetes playgrounds:iximiuz LabsKillercodaKodeKloudPlay with KubernetesYour Kubernetes server must be at or later than version v1.29.To check the version, enter kubectl version.Note:While you can use this feature with an earlier version, the feature is only GA and officially supported since v1.33.Extend Service IP RangesKubernetes clusters with kube-apiservers that have enabled the MultiCIDRServiceAllocator feature gate and have the networking.k8s.io/v1beta1 API group active, will create a ServiceCIDR object that takes the well-known name kubernetes, and that specifies an IP address range based on the value of the --service-cluster-ip-range command line argument to kube-apiserver.kubectl get servicecidr NAME CIDRS AGE kubernetes 10.96.0.0/28 17d The well-known kubernetes Service, that exposes the kube-apiserver endpoint to the Pods, calculates the first IP address from the default ServiceCIDR range and uses that IP address as its cluster IP address.kubectl get service kubernetes NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 17d The default Service, in this case, uses the ClusterIP 10.96.0.1, that has the corresponding IPAddress object.kubectl get ipaddress 10.96.0.1 NAME PARENTREF 10.96.0.1 services/default/kubernetes The ServiceCIDRs are protected with finalizers, to avoid leaving Service ClusterIPs orphans; the finalizer is only removed if there is another subnet that contains the existing IPAddresses or there are no IPAddresses belonging to the subnet.Extend the number of available IPs for ServicesThere are cases that users will need to increase the number addresses available to Services, previously, increasing the Service range was a disruptive operation that could also cause data loss. With this new feature users only need to add a new ServiceCIDR to increase the number of available addresses.Adding a new ServiceCIDROn a cluster with a 10.96.0.0/28 range for Services, there is only 2^(32-28) - 2 = 14 IP addresses available. The kubernetes.default Service is always created; for this example, that leaves you with only 13 possible Services.for i in $(seq 1 13); do kubectl create service clusterip "test-$i" --tcp 80 -o json | jq -r .spec.clusterIP; done 10.96.0.11 10.96.0.5 10.96.0.12 10.96.0.13 10.96.0.14 10.96.0.2 10.96.0.3 10.96.0.4 10.96.0.6 10.96.0.7 10.96.0.8 10.96.0.9 error: failed to create ClusterIP service: Internal error occurred: failed to allocate a serviceIP: range is full You can increase the number of IP addresses available for Services, by creating a new ServiceCIDR that extends or adds new IP address ranges.cat <EOF | kubectl apply -f - apiVersion: networking.k8s.io/v1beta1 kind: ServiceCIDR metadata: name: newcidr1 spec: cidrs: - 10.96.0.0/24 EOF servicecidr.networking.k8s.io/newcidr1 created and this will allow you to create new Services with ClusterIPs that will be picked from this new range.for i in $(seq 13 16); do kubectl create service clusterip "test-$i" --tcp 80 -o json | jq -r .spec.clusterIP; done 10.96.0.48 10.96.0.200 10.96.0.121 10.96.0.144 Deleting a ServiceCIDRYou cannot delete a ServiceCIDR if there are IPAddresses that depend on the ServiceCIDR.kubectl delete servicecidr newcidr1 servicecidr.networking.k8s.io "newcidr1" deleted Kubernetes uses a finalizer on the ServiceCIDR to track this dependent relationship.kubectl get servicecidr newcidr1 -o yaml apiVersion: networking.k8s.io/v1beta1 kind: ServiceCIDR metadata: creationTimestamp: "2023-10-12T15:11:07Z" deletionGracePeriodSeconds: 0 deletionTimestamp: "2023-10-12T15:12:45Z" finalizers: - networking.k8s.io/service-cidr-finalizer name: newcidr1 resourceVersion: "1133" uid: 5ffd8afe-c78f-4e60-ae76-cec448a8af40 spec: cidrs: - 10.96.0.0/24 status: conditions: - lastTransitionTime: "2023-10-12T15:12:45Z" message: There are still IPAddresses referencing the ServiceCIDR, please remove them or create a new ServiceCIDR reason: OrphanIPAddress status: "False" type: Ready By removing the Services containing the IP addresses that are blocking the deletion of the ServiceCIDRfor i in $(seq 13 16); do kubectl delete service "test-$i" ; done service "test-13" deleted service "test-14" deleted service "test-15" deleted service "test-16" deleted the control plane notices the removal. The control plane then removes its finalizer, so that the ServiceCIDR that was pending deletion will actually be removed.kubectl get servicecidr newcidr1 Error from server (NotFound): servicecidrs.networking.k8s.io "newcidr1" not found Kubernetes Service CIDR PoliciesCluster administrators can implement policies to control the creation and modification of ServiceCIDR resources within the cluster. This allows for centralized management of the IP address ranges used for Services and helps prevent unintended or conflicting configurations. Kubernetes provides mechanisms like Validating Admission Policies to enforce these rules.Preventing Unauthorized ServiceCIDR Creation/Update using Validating Admission PolicyThere can be situations that the cluster administrators want to restrict the ranges that can be allowed or to completely deny any changes to the cluster Service IP ranges.Note:The default "kubernetes" ServiceCIDR is created by the kube-apiserver to provide consistency in the cluster and is required for the cluster to work, so it always must be allowed. You can ensure your ValidatingAdmissionPolicy doesn't restrict the default ServiceCIDR by adding the clause: matchConditions: - name: 'exclude-default-servicecidr' expression: "object.metadata.name != 'kubernetes'" as in the examples below.Restrict Service CIDR ranges to some specific rangesThe following is an example of a ValidatingAdmissionPolicy that only allows ServiceCIDRs to be created if they are subranges of the given allowed ranges. (So the example policy would allow a ServiceCIDR with cidrs: ['10.96.1.0/24'] or cidrs: ['2001:db8:0:0:ffff::/80', '10.96.0.0/20'] but would not allow a ServiceCIDR with cidrs: ['172.20.0.0/16'].) You can copy this policy and change the value of allowed to something appropriate for you cluster.apiVersion: admissionregistration.k8s.io/v1 kind: ValidatingAdmissionPolicy metadata: name: "servicecidrs.default" spec: failurePolicy: Fail matchConstraints: resourceRules: - apiGroups: ["networking.k8s.io"] apiVersions: ["v1","v1beta1"] operations: ["CREATE", "UPDATE"] resources: ["servicecidrs"] matchConditions: - name: 'exclude-default-servicecidr' expression: "object.metadata.name != 'kubernetes'" variables: - name: allowed expression: "['10.96.0.0/16','2001:db8::/64']" validations: - expression: "object.spec.cidrs.all(newCIDR, variables.allowed.exists(allowedCIDR, cidr(allowedCIDR).containsCIDR(newCIDR)))" # For all CIDRs (newCIDR) listed in the spec.cidrs of the submitted ServiceCIDR # object, check if there exists at least one CIDR (allowedCIDR) in the `allowed` # list of the VAP such that the allowedCIDR fully contains the newCIDR. --- apiVersion: admissionregistration.k8s.io/v1 kind: ValidatingAdmissionPolicyBinding metadata: name: "servicecidrs-binding" spec: policyName: "servicecidrs.default" validationActions: [Deny,Audit] Consult the CEL documentation to learn more about CEL if you want to write your own validation expression.Restrict any usage of the ServiceCIDR APIThe following example demonstrates how to use a ValidatingAdmissionPolicy and its binding to restrict the creation of any new Service CIDR ranges, excluding the default "kubernetes" ServiceCIDR:apiVersion: admissionregistration.k8s.io/v1 kind: ValidatingAdmissionPolicy metadata: name: "servicecidrs.deny" spec: failurePolicy: Fail matchConstraints: resourceRules: - apiGroups: ["networking.k8s.io"] apiVersions: ["v1","v1beta1"] operations: ["CREATE", "UPDATE"] resources: ["servicecidrs"] validations: - expression: "object.metadata.name == 'kubernetes'" --- apiVersion: admissionregistration.k8s.io/v1 kind: ValidatingAdmissionPolicyBinding metadata: name: "servicecidrs-deny-binding" spec: policyName: "servicecidrs.deny" validationActions: [Deny,Audit]

Extend Service IP Ranges


kubectl create jobSynopsisCreate a job with the specified name.kubectl create job NAME --image=image [--from=cronjob/name] -- [COMMAND] [args...] Examples # Create a job kubectl create job my-job --image=busybox # Create a job with a command kubectl create job my-job --image=busybox -- date # Create a job from a cron job named "a-cronjob" kubectl create job test-job --from=cronjob/a-cronjob Options--allow-missing-template-keys Default: trueIf true, ignore any errors in templates when a field or map key is missing in the template. Only applies to golang and jsonpath output formats.--dry-run string[="unchanged"] Default: "none"Must be "none", "server", or "client". If client strategy, only print the object that would be sent, without sending it. If server strategy, submit server-side request without persisting the resource.--field-manager string Default: "kubectl-create"Name of the manager used to track field ownership.--from stringThe name of the resource to create a Job from (only cronjob is supported).-h, --helphelp for job--image stringImage name to run.-o, --output stringOutput format. One of: (json, yaml, name, go-template, go-template-file, template, templatefile, jsonpath, jsonpath-as-json, jsonpath-file).--save-configIf true, the configuration of current object will be saved in its annotation. Otherwise, the annotation will be unchanged. This flag is useful when you want to perform kubectl apply on this object in the future.--show-managed-fieldsIf true, keep the managedFields when printing objects in JSON or YAML format.--template stringTemplate string or path to template file to use when -o=go-template, -o=go-template-file. The template format is golang templates [http://golang.org/pkg/text/template/#pkg-overview].--validate string[="strict"] Default: "strict"Must be one of: strict (or true), warn, ignore (or false). "true" or "strict" will use a schema to validate the input and fail the request if invalid. It will perform server side validation if ServerSideFieldValidation is enabled on the api-server, but will fall back to less reliable client-side validation if not. "warn" will warn about unknown or duplicate fields without blocking the request if server-side field validation is enabled on the API server, and behave as "ignore" otherwise. "false" or "ignore" will not perform any schema validation, silently dropping any unknown or duplicate fields.Parent Options Inherited--as stringUsername to impersonate for the operation. User could be a regular user or a service account in a namespace.--as-group stringsGroup to impersonate for the operation, this flag can be repeated to specify multiple groups.--as-uid stringUID to impersonate for the operation.--cache-dir string Default: "$HOME/.kube/cache"Default cache directory--certificate-authority stringPath to a cert file for the certificate authority--client-certificate stringPath to a client certificate file for TLS--client-key stringPath to a client key file for TLS--cluster stringThe name of the kubeconfig cluster to use--context stringThe name of the kubeconfig context to use--disable-compressionIf true, opt-out of response compression for all requests to the server--insecure-skip-tls-verifyIf true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure--kubeconfig stringPath to the kubeconfig file to use for CLI requests.--kuberc stringPath to the kuberc file to use for preferences. This can be disabled by exporting KUBECTL_KUBERC=false feature gate or turning off the feature KUBERC=off.--match-server-versionRequire server version to match client version-n, --namespace stringIf present, the namespace scope for this CLI request--password stringPassword for basic authentication to the API server--profile string Default: "none"Name of profile to capture. One of (none|cpu|heap|goroutine|threadcreate|block|mutex)--profile-output string Default: "profile.pprof"Name of the file to write the profile to--request-timeout string Default: "0"The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests.-s, --server stringThe address and port of the Kubernetes API server--storage-driver-buffer-duration duration Default: 1m0sWrites in the storage driver will be buffered for this duration, and committed to the non memory backends as a single transaction--storage-driver-db string Default: "cadvisor"database name--storage-driver-host string Default: "localhost:8086"database host:port--storage-driver-password string Default: "root"database password--storage-driver-secureuse secure connection with database--storage-driver-table string Default: "stats"table name--storage-driver-user string Default: "root"database username--tls-server-name stringServer name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used--token stringBearer token for authentication to the API server--user stringThe name of the kubeconfig user to use--username stringUsername for basic authentication to the API server--version version[=true]--version, --version=raw prints version information and quits; --version=vX.Y.Z... sets the reported version--warnings-as-errorsTreat warnings received from the server as errors and exit with a non-zero exit codeSee Alsokubectl create - Create a resource from a file or from stdinThis page is automatically generated.If you plan to report an issue with this page, mention that the page is auto-generated in your issue description. The fix may need to happen elsewhere in the Kubernetes project.

kubectl create job


Metrics for Kubernetes Object Stateskube-state-metrics, an add-on agent to generate and expose cluster-level metrics.The state of Kubernetes objects in the Kubernetes API can be exposed as metrics. An add-on agent called kube-state-metrics can connect to the Kubernetes API server and expose a HTTP endpoint with metrics generated from the state of individual objects in the cluster. It exposes various information about the state of objects like labels and annotations, startup and termination times, status or the phase the object currently is in. For example, containers running in pods create a kube_pod_container_info metric. This includes the name of the container, the name of the pod it is part of, the namespace the pod is running in, the name of the container image, the ID of the image, the image name from the spec of the container, the ID of the running container and the ID of the pod as labels.🛇 This item links to a third party project or product that is not part of Kubernetes itself. More informationAn external component that is able and capable to scrape the endpoint of kube-state-metrics (for example via Prometheus) can now be used to enable the following use cases.Example: using metrics from kube-state-metrics to query the cluster stateMetric series generated by kube-state-metrics are helpful to gather further insights into the cluster, as they can be used for querying.If you use Prometheus or another tool that uses the same query language, the following PromQL query returns the number of pods that are not ready:count(kube_pod_status_ready{condition="false"}) by (namespace, pod) Example: alerting based on from kube-state-metricsMetrics generated from kube-state-metrics also allow for alerting on issues in the cluster.If you use Prometheus or a similar tool that uses the same alert rule language, the following alert will fire if there are pods that have been in a Terminating state for more than 5 minutes:groups: - name: Pod state rules: - alert: PodsBlockedInTerminatingState expr: count(kube_pod_deletion_timestamp) by (namespace, pod) * count(kube_pod_status_reason{reason="NodeLost"} == 0) by (namespace, pod) > 0 for: 5m labels: severity: page annotations: summary: Pod {{$labels.namespace}}/{{$labels.pod}} blocked in Terminating state. Items on this page refer to third party products or projects that provide functionality required by Kubernetes. The Kubernetes project authors aren't responsible for those third-party products or projects. See the CNCF website guidelines for more details.You should read the content guide before proposing a change that adds an extra third-party link.

Metrics for Kubernetes Object States


AuthorizationDetails of Kubernetes authorization mechanisms and supported authorization modes.Kubernetes authorization takes place following authentication. Usually, a client making a request must be authenticated (logged in) before its request can be allowed; however, Kubernetes also allows anonymous requests in some circumstances.For an overview of how authorization fits into the wider context of API access control, read Controlling Access to the Kubernetes API.Authorization verdictsKubernetes authorization of API requests takes place within the API server. The API server evaluates all of the request attributes against all policies, potentially also consulting external services, and then allows or denies the request.All parts of an API request must be allowed by some authorization mechanism in order to proceed. In other words: access is denied by default.Note:Access controls and policies that depend on specific fields of specific kinds of objects are handled by admission controllers.Kubernetes admission control happens after authorization has completed (and, therefore, only when the authorization decision was to allow the request).When multiple authorization modules are configured, each is checked in sequence. If any authorizer approves or denies a request, that decision is immediately returned and no other authorizer is consulted. If all modules have no opinion on the request, then the request is denied. An overall deny verdict means that the API server rejects the request and responds with an HTTP 403 (Forbidden) status.Request attributes used in authorizationKubernetes reviews only the following API request attributes:user - The user string provided during authentication.group - The list of group names to which the authenticated user belongs.extra - A map of arbitrary string keys to string values, provided by the authentication layer.API - Indicates whether the request is for an API resource.Request path - Path to miscellaneous non-resource endpoints like /api or /healthz.API request verb - API verbs like get, list, create, update, patch, watch, delete, and deletecollection are used for resource requests. To determine the request verb for a resource API endpoint, see request verbs and authorization.HTTP request verb - Lowercased HTTP methods like get, post, put, and delete are used for non-resource requests.Resource - The ID or name of the resource that is being accessed (for resource requests only) -- For resource requests using get, update, patch, and delete verbs, you must provide the resource name.Subresource - The subresource that is being accessed (for resource requests only).Namespace - The namespace of the object that is being accessed (for namespaced resource requests only).API group - The API Group being accessed (for resource requests only). An empty string designates the core API group.Request verbs and authorizationNon-resource requestsRequests to endpoints other than /api/v1/... or /apis/<group>/<version>/... are considered non-resource requests, and use the lower-cased HTTP method of the request as the verb. For example, making a GET request using HTTP to endpoints such as /api or /healthz would use get as the verb.Resource requestsTo determine the request verb for a resource API endpoint, Kubernetes maps the HTTP verb used and considers whether or not the request acts on an individual resource or on a collection of resources:HTTP verbrequest verbPOSTcreateGET, HEADget (for individual resources), list (for collections, including full object content), watch (for watching an individual resource or collection of resources)PUTupdatePATCHpatchDELETEdelete (for individual resources), deletecollection (for collections)Caution:+The get, list and watch verbs can all return the full details of a resource. In terms of access to the returned data they are equivalent. For example, list on secrets will reveal the data attributes of any returned resources.Kubernetes sometimes checks authorization for additional permissions using specialized verbs. For example:Special cases of authenticationimpersonate verb on users, groups, and serviceaccounts in the core API group, and the userextras in the authentication.k8s.io API group.Authorization of CertificateSigningRequestsapprove verb for CertificateSigningRequests, and update for revisions to existing approvalsRBACbind and escalate verbs on roles and clusterroles resources in the rbac.authorization.k8s.io API group.Authorization contextKubernetes expects attributes that are common to REST API requests. This means that Kubernetes authorization works with existing organization-wide or cloud-provider-wide access control systems which may handle other APIs besides the Kubernetes API.Authorization modesThe Kubernetes API server may authorize a request using one of several authorization modes:AlwaysAllowThis mode allows all requests, which brings security risks. Use this authorization mode only if you do not require authorization for your API requests (for example, for testing).AlwaysDenyThis mode blocks all requests. Use this authorization mode only for testing.ABAC (attribute-based access control)Kubernetes ABAC mode defines an access control paradigm whereby access rights are granted to users through the use of policies which combine attributes together. The policies can use any type of attributes (user attributes, resource attributes, object, environment attributes, etc).RBAC (role-based access control)Kubernetes RBAC is a method of regulating access to computer or network resources based on the roles of individual users within an enterprise. In this context, access is the ability of an individual user to perform a specific task, such as view, create, or modify a file.In this mode, Kubernetes uses the rbac.authorization.k8s.io API group to drive authorization decisions, allowing you to dynamically configure permission policies through the Kubernetes API.NodeA special-purpose authorization mode that grants permissions to kubelets based on the pods they are scheduled to run. To learn more about the Node authorization mode, see Node Authorization.WebhookKubernetes webhook mode for authorization makes a synchronous HTTP callout, blocking the request until the remote HTTP service responds to the query.You can write your own software to handle the callout, or use solutions from the ecosystem.Warning:Enabling the AlwaysAllow mode bypasses authorization; do not use this on a cluster where you do not trust all potential API clients, including the workloads that you run.Authorization mechanisms typically return either a deny or no opinion result; see authorization verdicts for more on this. Activating the AlwaysAllow means that if all other authorizers return “no opinion”, the request is allowed. For example, --authorization-mode=AlwaysAllow,RBAC has the same effect as --authorization-mode=AlwaysAllow because Kubernetes RBAC does not provide negative (deny) access rules.You should not use the AlwaysAllow mode on a Kubernetes cluster where the API server is reachable from the public internet.The system:masters groupThe system:masters group is a built-in Kubernetes group that grants unrestricted access to the API server. Any user assigned to this group has full cluster administrator privileges, bypassing any authorization restrictions imposed by the RBAC or Webhook mechanisms. Avoid adding users to this group. If you do need to grant a user cluster-admin rights, you can create a ClusterRoleBinding to the built-in cluster-admin ClusterRole.Authorization mode configurationYou can configure the Kubernetes API server's authorizer chain using either a configuration file only or command line arguments.You have to pick one of the two configuration approaches; setting both --authorization-config path and configuring an authorization webhook using the --authorization-mode and --authorization-webhook-* command line arguments is not allowed. If you try this, the API server reports an error message during startup, then exits immediately.Configuring the API Server using an authorization config fileFEATURE STATE: Kubernetes v1.32 [stable] (enabled by default: true)Kubernetes lets you configure authorization chains that can include multiple webhooks. The authorization items in that chain can have well-defined parameters that validate requests in a particular order, offering you fine-grained control, such as explicit Deny on failures.The configuration file approach even allows you to specify CEL rules to pre-filter requests before they are dispatched to webhooks, helping you to prevent unnecessary invocations. The API server also automatically reloads the authorizer chain when the configuration file is modified.You specify the path to the authorization configuration using the --authorization-config command line argument.If you want to use command line arguments instead of a configuration file, that's also a valid and supported approach. Some authorization capabilities (for example: multiple webhooks, webhook failure policy, and pre-filter rules) are only available if you use an authorization configuration file.Example configuration--- # # DO NOT USE THE CONFIG AS IS. THIS IS AN EXAMPLE. # apiVersion: apiserver.config.k8s.io/v1 kind: AuthorizationConfiguration authorizers: - type: Webhook # Name used to describe the authorizer # This is explicitly used in monitoring machinery for metrics # Note: # - Validation for this field is similar to how K8s labels are validated today. # Required, with no default name: webhook webhook: # The duration to cache 'authorized' responses from the webhook # authorizer. # Same as setting `--authorization-webhook-cache-authorized-ttl` flag # Default: 5m0s authorizedTTL: 30s # If set to false, 'authorized' responses from the webhook are not cached # and the specified authorizedTTL is ignored/has no effect. # Same as setting `--authorization-webhook-cache-authorized-ttl` flag to `0`. # Note: Setting authorizedTTL to `0` results in its default value being used. # Default: true cacheAuthorizedRequests: true # The duration to cache 'unauthorized' responses from the webhook # authorizer. # Same as setting `--authorization-webhook-cache-unauthorized-ttl` flag # Default: 30s unauthorizedTTL: 30s # If set to false, 'unauthorized' responses from the webhook are not cached # and the specified unauthorizedTTL is ignored/has no effect. # Same as setting `--authorization-webhook-cache-unauthorized-ttl` flag to `0`. # Note: Setting unauthorizedTTL to `0` results in its default value being used. # Default: true cacheUnauthorizedRequests: true # Timeout for the webhook request # Maximum allowed is 30s. # Required, with no default. timeout: 3s # The API version of the authorization.k8s.io SubjectAccessReview to # send to and expect from the webhook. # Same as setting `--authorization-webhook-version` flag # Required, with no default # Valid values: v1beta1, v1 subjectAccessReviewVersion: v1 # MatchConditionSubjectAccessReviewVersion specifies the SubjectAccessReview # version the CEL expressions are evaluated against # Valid values: v1 # Required, no default value matchConditionSubjectAccessReviewVersion: v1 # Controls the authorization decision when a webhook request fails to # complete or returns a malformed response or errors evaluating # matchConditions. # Valid values: # - NoOpinion: continue to subsequent authorizers to see if one of # them allows the request # - Deny: reject the request without consulting subsequent authorizers # Required, with no default. failurePolicy: Deny connectionInfo: # Controls how the webhook should communicate with the server. # Valid values: # - KubeConfigFile: use the file specified in kubeConfigFile to locate the # server. # - InClusterConfig: use the in-cluster configuration to call the # SubjectAccessReview API hosted by kube-apiserver. This mode is not # allowed for kube-apiserver. type: KubeConfigFile # Path to KubeConfigFile for connection info # Required, if connectionInfo.Type is KubeConfigFile kubeConfigFile: /kube-system-authz-webhook.yaml # matchConditions is a list of conditions that must be met for a request to be sent to this # webhook. An empty list of matchConditions matches all requests. # There are a maximum of 64 match conditions allowed. # # The exact matching logic is (in order): # 1. If at least one matchCondition evaluates to FALSE, then the webhook is skipped. # 2. If ALL matchConditions evaluate to TRUE, then the webhook is called. # 3. If at least one matchCondition evaluates to an error (but none are FALSE): # - If failurePolicy=Deny, then the webhook rejects the request # - If failurePolicy=NoOpinion, then the error is ignored and the webhook is skipped matchConditions: # expression represents the expression which will be evaluated by CEL. Must evaluate to bool. # CEL expressions have access to the contents of the SubjectAccessReview in v1 version. # If version specified by subjectAccessReviewVersion in the request variable is v1beta1, # the contents would be converted to the v1 version before evaluating the CEL expression. # # Documentation on CEL: https://kubernetes.io/docs/reference/using-api/cel/ # # only send resource requests to the webhook - expression: has(request.resourceAttributes) # only intercept requests to kube-system - expression: request.resourceAttributes.namespace == 'kube-system' # don't intercept requests from kube-system service accounts - expression: "!('system:serviceaccounts:kube-system' in request.groups)" - type: Node name: node - type: RBAC name: rbac - type: Webhook name: in-cluster-authorizer webhook: authorizedTTL: 5m unauthorizedTTL: 30s timeout: 3s subjectAccessReviewVersion: v1 failurePolicy: NoOpinion connectionInfo: type: InClusterConfigWhen configuring the authorizer chain using a configuration file, make sure all the control plane nodes have the same file contents. Take a note of the API server configuration when upgrading / downgrading your clusters. For example, if upgrading from Kubernetes 1.33 to Kubernetes 1.34, you would need to make sure the config file is in a format that Kubernetes 1.34 can understand, before you upgrade the cluster. If you downgrade to 1.33, you would need to set the configuration appropriately.Authorization configuration and reloadsKubernetes reloads the authorization configuration file when the API server observes a change to the file, and also on a 60 second schedule if no change events were observed.Note:You must ensure that all non-webhook authorizer types remain unchanged in the file on reload.A reload must not add or remove Node or RBAC authorizers (they can be reordered, but cannot be added or removed).Command line authorization mode configurationYou can use the following modes:--authorization-mode=ABAC (Attribute-based access control mode)--authorization-mode=RBAC (Role-based access control mode)--authorization-mode=Node (Node authorizer)--authorization-mode=Webhook (Webhook authorization mode)--authorization-mode=AlwaysAllow (always allows requests; carries security risks)--authorization-mode=AlwaysDeny (always denies requests)You can choose more than one authorization mode; for example: --authorization-mode=Node,RBAC,WebhookKubernetes checks authorization modules based on the order that you specify them on the API server's command line, so an earlier module has higher priority to allow or deny a request.You cannot combine the --authorization-mode command line argument with the --authorization-config command line argument used for configuring authorization using a local file.For more information on command line arguments to the API server, read the kube-apiserver reference.Privilege escalation via workload creation or editsUsers who can create/edit pods in a namespace, either directly or through an object that enables indirect workload management, may be able to escalate their privileges in that namespace. The potential routes to privilege escalation include Kubernetes API extensions and their associated controllers.Caution:As a cluster administrator, use caution when granting access to create or edit workloads. Some details of how these can be misused are documented in escalation paths.Escalation pathsThere are different ways that an attacker or untrustworthy user could gain additional privilege within a namespace, if you allow them to run arbitrary Pods in that namespace:Mounting arbitrary Secrets in that namespaceCan be used to access confidential information meant for other workloadsCan be used to obtain a more privileged ServiceAccount's service account tokenUsing arbitrary ServiceAccounts in that namespaceCan perform Kubernetes API actions as another workload (impersonation)Can perform any privileged actions that ServiceAccount hasMounting or using ConfigMaps meant for other workloads in that namespaceCan be used to obtain information meant for other workloads, such as database host names.Mounting volumes meant for other workloads in that namespaceCan be used to obtain information meant for other workloads, and change it.Caution:As a system administrator, you should be cautious when deploying CustomResourceDefinitions that let users make changes to the above areas. These may open privilege escalations paths. Consider the consequences of this kind of change when deciding on your authorization controls.Checking API accesskubectl provides the auth can-i subcommand for quickly querying the API authorization layer. The command uses the SelfSubjectAccessReview API to determine if the current user can perform a given action, and works regardless of the authorization mode used.kubectl auth can-i create deployments --namespace dev The output is similar to this:yes kubectl auth can-i create deployments --namespace prod The output is similar to this:no Administrators can combine this with user impersonation to determine what action other users can perform.kubectl auth can-i list secrets --namespace dev --as dave The output is similar to this:no Similarly, to check whether a ServiceAccount named dev-sa in Namespace dev can list Pods in the Namespace target:kubectl auth can-i list pods \ --namespace target \ --as system:serviceaccount:dev:dev-sa The output is similar to this:yes SelfSubjectAccessReview is part of the authorization.k8s.io API group, which exposes the API server authorization to external services. Other resources in this group include:SubjectAccessReviewAccess review for any user, not only the current one. Useful for delegating authorization decisions to the API server. For example, the kubelet and extension API servers use this to determine user access to their own APIs.LocalSubjectAccessReviewLike SubjectAccessReview but restricted to a specific namespace.SelfSubjectRulesReviewA review which returns the set of actions a user can perform within a namespace. Useful for users to quickly summarize their own access, or for UIs to hide/show actions.These APIs can be queried by creating normal Kubernetes resources, where the response status field of the returned object is the result of the query. For example:kubectl create -f - -o yaml << EOF apiVersion: authorization.k8s.io/v1 kind: SelfSubjectAccessReview spec: resourceAttributes: group: apps resource: deployments verb: create namespace: dev EOF The generated SelfSubjectAccessReview is similar to:apiVersion: authorization.k8s.io/v1 kind: SelfSubjectAccessReview metadata: creationTimestamp: null spec: resourceAttributes: group: apps resource: deployments namespace: dev verb: create status: allowed: true denied: falseWhat's nextTo learn more about Authentication, see Authentication.For an overview, read Controlling Access to the Kubernetes API.To learn more about Admission Control, see Using Admission Controllers.Read more about Common Expression Language in Kubernetes.

Authorization