ClusterRoleClusterRole is a cluster level, logical grouping of PolicyRules that can be referenced as a unit by a RoleBinding or ClusterRoleBinding.apiVersion: rbac.authorization.k8s.io/v1import "k8s.io/api/rbac/v1"ClusterRoleClusterRole is a cluster level, logical grouping of PolicyRules that can be referenced as a unit by a RoleBinding or ClusterRoleBinding.apiVersion: rbac.authorization.k8s.io/v1kind: ClusterRolemetadata (ObjectMeta)Standard object's metadata.aggregationRule (AggregationRule)AggregationRule is an optional field that describes how to build the Rules for this ClusterRole. If AggregationRule is set, then the Rules are controller managed and direct changes to Rules will be stomped by the controller.AggregationRule describes how to locate ClusterRoles to aggregate into the ClusterRoleaggregationRule.clusterRoleSelectors ([]LabelSelector)Atomic: will be replaced during a mergeClusterRoleSelectors holds a list of selectors which will be used to find ClusterRoles and create the rules. If any of the selectors match, then the ClusterRole's permissions will be addedrules ([]PolicyRule)Atomic: will be replaced during a mergeRules holds all the PolicyRules for this ClusterRolePolicyRule holds information that describes a policy rule, but does not contain information about who the rule applies to or which namespace the rule applies to.rules.apiGroups ([]string)Atomic: will be replaced during a mergeAPIGroups is the name of the APIGroup that contains the resources. If multiple API groups are specified, any action requested against one of the enumerated resources in any API group will be allowed. "" represents the core API group and "*" represents all API groups.rules.resources ([]string)Atomic: will be replaced during a mergeResources is a list of resources this rule applies to. '*' represents all resources.rules.verbs ([]string), requiredAtomic: will be replaced during a mergeVerbs is a list of Verbs that apply to ALL the ResourceKinds contained in this rule. '*' represents all verbs.rules.resourceNames ([]string)Atomic: will be replaced during a mergeResourceNames is an optional white list of names that the rule applies to. An empty set means that everything is allowed.rules.nonResourceURLs ([]string)Atomic: will be replaced during a mergeNonResourceURLs is a set of partial urls that a user should have access to. *s are allowed, but only as the full, final step in the path Since non-resource URLs are not namespaced, this field is only applicable for ClusterRoles referenced from a ClusterRoleBinding. Rules can either apply to API resources (such as "pods" or "secrets") or non-resource URL paths (such as "/api"), but not both.ClusterRoleListClusterRoleList is a collection of ClusterRolesapiVersion: rbac.authorization.k8s.io/v1kind: ClusterRoleListmetadata (ListMeta)Standard object's metadata.items ([]ClusterRole), requiredItems is a list of ClusterRolesOperationsget read the specified ClusterRoleHTTP RequestGET /apis/rbac.authorization.k8s.io/v1/clusterroles/{name}Parametersname (in path): string, requiredname of the ClusterRolepretty (in query): stringprettyResponse200 (ClusterRole): OK401: Unauthorizedlist list or watch objects of kind ClusterRoleHTTP RequestGET /apis/rbac.authorization.k8s.io/v1/clusterrolesParametersallowWatchBookmarks (in query): booleanallowWatchBookmarkscontinue (in query): stringcontinuefieldSelector (in query): stringfieldSelectorlabelSelector (in query): stringlabelSelectorlimit (in query): integerlimitpretty (in query): stringprettyresourceVersion (in query): stringresourceVersionresourceVersionMatch (in query): stringresourceVersionMatchsendInitialEvents (in query): booleansendInitialEventstimeoutSeconds (in query): integertimeoutSecondswatch (in query): booleanwatchResponse200 (ClusterRoleList): OK401: Unauthorizedcreate create a ClusterRoleHTTP RequestPOST /apis/rbac.authorization.k8s.io/v1/clusterrolesParametersbody: ClusterRole, requireddryRun (in query): stringdryRunfieldManager (in query): stringfieldManagerfieldValidation (in query): stringfieldValidationpretty (in query): stringprettyResponse200 (ClusterRole): OK201 (ClusterRole): Created202 (ClusterRole): Accepted401: Unauthorizedupdate replace the specified ClusterRoleHTTP RequestPUT /apis/rbac.authorization.k8s.io/v1/clusterroles/{name}Parametersname (in path): string, requiredname of the ClusterRolebody: ClusterRole, requireddryRun (in query): stringdryRunfieldManager (in query): stringfieldManagerfieldValidation (in query): stringfieldValidationpretty (in query): stringprettyResponse200 (ClusterRole): OK201 (ClusterRole): Created401: Unauthorizedpatch partially update the specified ClusterRoleHTTP RequestPATCH /apis/rbac.authorization.k8s.io/v1/clusterroles/{name}Parametersname (in path): string, requiredname of the ClusterRolebody: Patch, requireddryRun (in query): stringdryRunfieldManager (in query): stringfieldManagerfieldValidation (in query): stringfieldValidationforce (in query): booleanforcepretty (in query): stringprettyResponse200 (ClusterRole): OK201 (ClusterRole): Created401: Unauthorizeddelete delete a ClusterRoleHTTP RequestDELETE /apis/rbac.authorization.k8s.io/v1/clusterroles/{name}Parametersname (in path): string, requiredname of the ClusterRolebody: DeleteOptionsdryRun (in query): stringdryRungracePeriodSeconds (in query): integergracePeriodSecondsignoreStoreReadErrorWithClusterBreakingPotential (in query): booleanignoreStoreReadErrorWithClusterBreakingPotentialpretty (in query): stringprettypropagationPolicy (in query): stringpropagationPolicyResponse200 (Status): OK202 (Status): Accepted401: Unauthorizeddeletecollection delete collection of ClusterRoleHTTP RequestDELETE /apis/rbac.authorization.k8s.io/v1/clusterrolesParametersbody: DeleteOptionscontinue (in query): stringcontinuedryRun (in query): stringdryRunfieldSelector (in query): stringfieldSelectorgracePeriodSeconds (in query): integergracePeriodSecondsignoreStoreReadErrorWithClusterBreakingPotential (in query): booleanignoreStoreReadErrorWithClusterBreakingPotentiallabelSelector (in query): stringlabelSelectorlimit (in query): integerlimitpretty (in query): stringprettypropagationPolicy (in query): stringpropagationPolicyresourceVersion (in query): stringresourceVersionresourceVersionMatch (in query): stringresourceVersionMatchsendInitialEvents (in query): booleansendInitialEventstimeoutSeconds (in query): integertimeoutSecondsResponse200 (Status): OK401: UnauthorizedThis page is automatically generated.If you plan to report an issue with this page, mention that the page is auto-generated in your issue description. The fix may need to happen elsewhere in the Kubernetes project.

ClusterRole


Owners and DependentsIn Kubernetes, some objects are owners of other objects. For example, a ReplicaSet is the owner of a set of Pods. These owned objects are dependents of their owner.Ownership is different from the labels and selectors mechanism that some resources also use. For example, consider a Service that creates EndpointSlice objects. The Service uses labels to allow the control plane to determine which EndpointSlice objects are used for that Service. In addition to the labels, each EndpointSlice that is managed on behalf of a Service has an owner reference. Owner references help different parts of Kubernetes avoid interfering with objects they don’t control.Owner references in object specificationsDependent objects have a metadata.ownerReferences field that references their owner object. A valid owner reference consists of the object name and a UID within the same namespace as the dependent object. Kubernetes sets the value of this field automatically for objects that are dependents of other objects like ReplicaSets, DaemonSets, Deployments, Jobs and CronJobs, and ReplicationControllers. You can also configure these relationships manually by changing the value of this field. However, you usually don't need to and can allow Kubernetes to automatically manage the relationships.Dependent objects also have an ownerReferences.blockOwnerDeletion field that takes a boolean value and controls whether specific dependents can block garbage collection from deleting their owner object. Kubernetes automatically sets this field to true if a controller (for example, the Deployment controller) sets the value of the metadata.ownerReferences field. You can also set the value of the blockOwnerDeletion field manually to control which dependents block garbage collection.A Kubernetes admission controller controls user access to change this field for dependent resources, based on the delete permissions of the owner. This control prevents unauthorized users from delaying owner object deletion.Note:Cross-namespace owner references are disallowed by design. Namespaced dependents can specify cluster-scoped or namespaced owners. A namespaced owner must exist in the same namespace as the dependent. If it does not, the owner reference is treated as absent, and the dependent is subject to deletion once all owners are verified absent.Cluster-scoped dependents can only specify cluster-scoped owners. In v1.20+, if a cluster-scoped dependent specifies a namespaced kind as an owner, it is treated as having an unresolvable owner reference, and is not able to be garbage collected.In v1.20+, if the garbage collector detects an invalid cross-namespace ownerReference, or a cluster-scoped dependent with an ownerReference referencing a namespaced kind, a warning Event with a reason of OwnerRefInvalidNamespace and an involvedObject of the invalid dependent is reported. You can check for that kind of Event by running kubectl get events -A --field-selector=reason=OwnerRefInvalidNamespace.Ownership and finalizersWhen you tell Kubernetes to delete a resource, the API server allows the managing controller to process any finalizer rules for the resource. Finalizers prevent accidental deletion of resources your cluster may still need to function correctly. For example, if you try to delete a PersistentVolume that is still in use by a Pod, the deletion does not happen immediately because the PersistentVolume has the kubernetes.io/pv-protection finalizer on it. Instead, the volume remains in the Terminating status until Kubernetes clears the finalizer, which only happens after the PersistentVolume is no longer bound to a Pod.Kubernetes also adds finalizers to an owner resource when you use either foreground or orphan cascading deletion. In foreground deletion, it adds the foreground finalizer so that the controller must delete dependent resources that also have ownerReferences.blockOwnerDeletion=true before it deletes the owner. If you specify an orphan deletion policy, Kubernetes adds the orphan finalizer so that the controller ignores dependent resources after it deletes the owner object.What's nextLearn more about Kubernetes finalizers.Learn about garbage collection.Read the API reference for object metadata.

Owners and Dependents


Device PluginsDevice plugins let you configure your cluster with support for devices or resources that require vendor-specific setup, such as GPUs, NICs, FPGAs, or non-volatile main memory.FEATURE STATE: Kubernetes v1.26 [stable]Kubernetes provides a device plugin framework that you can use to advertise system hardware resources to the Kubelet.Instead of customizing the code for Kubernetes itself, vendors can implement a device plugin that you deploy either manually or as a DaemonSet. The targeted devices include GPUs, high-performance NICs, FPGAs, InfiniBand adapters, and other similar computing resources that may require vendor specific initialization and setup.Device plugin registrationThe kubelet exports a Registration gRPC service:service Registration { rpc Register(RegisterRequest) returns (Empty) {} } A device plugin can register itself with the kubelet through this gRPC service. During the registration, the device plugin needs to send:The name of its Unix socket.The Device Plugin API version against which it was built.The ResourceName it wants to advertise. Here ResourceName needs to follow the extended resource naming scheme as vendor-domain/resourcetype. (For example, an NVIDIA GPU is advertised as nvidia.com/gpu.)Following a successful registration, the device plugin sends the kubelet the list of devices it manages, and the kubelet is then in charge of advertising those resources to the API server as part of the kubelet node status update. For example, after a device plugin registers hardware-vendor.example/foo with the kubelet and reports two healthy devices on a node, the node status is updated to advertise that the node has 2 "Foo" devices installed and available.Then, users can request devices as part of a Pod specification (see container). Requesting extended resources is similar to how you manage requests and limits for other resources, with the following differences:Extended resources are only supported as integer resources and cannot be overcommitted.Devices cannot be shared between containers.ExampleSuppose a Kubernetes cluster is running a device plugin that advertises resource hardware-vendor.example/foo on certain nodes. Here is an example of a pod requesting this resource to run a demo workload:--- apiVersion: v1 kind: Pod metadata: name: demo-pod spec: containers: - name: demo-container-1 image: registry.k8s.io/pause:3.8 resources: limits: hardware-vendor.example/foo: 2 # # This Pod needs 2 of the hardware-vendor.example/foo devices # and can only schedule onto a Node that's able to satisfy # that need. # # If the Node has more than 2 of those devices available, the # remainder would be available for other Pods to use. Device plugin implementationThe general workflow of a device plugin includes the following steps:Initialization. During this phase, the device plugin performs vendor-specific initialization and setup to make sure the devices are in a ready state.The plugin starts a gRPC service, with a Unix socket under the host path /var/lib/kubelet/device-plugins/, that implements the following interfaces:service DevicePlugin { // GetDevicePluginOptions returns options to be communicated with Device Manager. rpc GetDevicePluginOptions(Empty) returns (DevicePluginOptions) {} // ListAndWatch returns a stream of List of Devices // Whenever a Device state change or a Device disappears, ListAndWatch // returns the new list rpc ListAndWatch(Empty) returns (stream ListAndWatchResponse) {} // Allocate is called during container creation so that the Device // Plugin can run device specific operations and instruct Kubelet // of the steps to make the Device available in the container rpc Allocate(AllocateRequest) returns (AllocateResponse) {} // GetPreferredAllocation returns a preferred set of devices to allocate // from a list of available ones. The resulting preferred allocation is not // guaranteed to be the allocation ultimately performed by the // devicemanager. It is only designed to help the devicemanager make a more // informed allocation decision when possible. rpc GetPreferredAllocation(PreferredAllocationRequest) returns (PreferredAllocationResponse) {} // PreStartContainer is called, if indicated by Device Plugin during registration phase, // before each container start. Device plugin can run device specific operations // such as resetting the device before making devices available to the container. rpc PreStartContainer(PreStartContainerRequest) returns (PreStartContainerResponse) {} } Note:Plugins are not required to provide useful implementations for GetPreferredAllocation() or PreStartContainer(). Flags indicating the availability of these calls, if any, should be set in the DevicePluginOptions message sent back by a call to GetDevicePluginOptions(). The kubelet will always call GetDevicePluginOptions() to see which optional functions are available, before calling any of them directly.The plugin registers itself with the kubelet through the Unix socket at host path /var/lib/kubelet/device-plugins/kubelet.sock.Note:The ordering of the workflow is important. A plugin MUST start serving gRPC service before registering itself with kubelet for successful registration.After successfully registering itself, the device plugin runs in serving mode, during which it keeps monitoring device health and reports back to the kubelet upon any device state changes. It is also responsible for serving Allocate gRPC requests. During Allocate, the device plugin may do device-specific preparation; for example, GPU cleanup or QRNG initialization. If the operations succeed, the device plugin returns an AllocateResponse that contains container runtime configurations for accessing the allocated devices. The kubelet passes this information to the container runtime.An AllocateResponse contains zero or more ContainerAllocateResponse objects. In these, the device plugin defines modifications that must be made to a container's definition to provide access to the device. These modifications include:Annotationsdevice nodesenvironment variablesmountsfully-qualified CDI device namesNote:The processing of the fully-qualified CDI device names by the Device Manager requires that the DevicePluginCDIDevices feature gate is enabled for both the kubelet and the kube-apiserver. This was added as an alpha feature in Kubernetes v1.28, graduated to beta in v1.29 and to GA in v1.31.Handling kubelet restartsA device plugin is expected to detect kubelet restarts and re-register itself with the new kubelet instance. A new kubelet instance deletes all the existing Unix sockets under /var/lib/kubelet/device-plugins when it starts. A device plugin can monitor the deletion of its Unix socket and re-register itself upon such an event.Device plugin and unhealthy devicesThere are cases when devices fail or are shut down. The responsibility of the Device Plugin in this case is to notify the kubelet about the situation using the ListAndWatchResponse API.Once a device is marked as unhealthy, the kubelet will decrease the allocatable count for this resource on the Node to reflect how many devices can be used for scheduling new pods. Capacity count for the resource will not change.Pods that were assigned to the failed devices will continue be assigned to this device. It is typical that code relying on the device will start failing and Pod may get into Failed phase if restartPolicy for the Pod was not Always or enter the crash loop otherwise.Before Kubernetes v1.31, the way to know whether or not a Pod is associated with the failed device is to use the PodResources API.FEATURE STATE: Kubernetes v1.31 [alpha] (enabled by default: false)By enabling the feature gate ResourceHealthStatus, the field allocatedResourcesStatus will be added to each container status, within the .status for each Pod. The allocatedResourcesStatus field reports health information for each device assigned to the container.For a failed Pod, or where you suspect a fault, you can use this status to understand whether the Pod behavior may be associated with device failure. For example, if an accelerator is reporting an over-temperature event, the allocatedResourcesStatus field may be able to report this.Device plugin deploymentYou can deploy a device plugin as a DaemonSet, as a package for your node's operating system, or manually.The canonical directory /var/lib/kubelet/device-plugins requires privileged access, so a device plugin must run in a privileged security context. If you're deploying a device plugin as a DaemonSet, /var/lib/kubelet/device-plugins must be mounted as a Volume in the plugin's PodSpec.If you choose the DaemonSet approach you can rely on Kubernetes to: place the device plugin's Pod onto Nodes, to restart the daemon Pod after failure, and to help automate upgrades.API compatibilityPreviously, the versioning scheme required the Device Plugin's API version to match exactly the Kubelet's version. Since the graduation of this feature to Beta in v1.12 this is no longer a hard requirement. The API is versioned and has been stable since Beta graduation of this feature. Because of this, kubelet upgrades should be seamless but there still may be changes in the API before stabilization making upgrades not guaranteed to be non-breaking.Note:Although the Device Manager component of Kubernetes is a generally available feature, the device plugin API is not stable. For information on the device plugin API and version compatibility, read Device Plugin API versions.As a project, Kubernetes recommends that device plugin developers:Watch for Device Plugin API changes in the future releases.Support multiple versions of the device plugin API for backward/forward compatibility.To run device plugins on nodes that need to be upgraded to a Kubernetes release with a newer device plugin API version, upgrade your device plugins to support both versions before upgrading these nodes. Taking that approach will ensure the continuous functioning of the device allocations during the upgrade.Monitoring device plugin resourcesFEATURE STATE: Kubernetes v1.28 [stable]In order to monitor resources provided by device plugins, monitoring agents need to be able to discover the set of devices that are in-use on the node and obtain metadata to describe which container the metric should be associated with. Prometheus metrics exposed by device monitoring agents should follow the Kubernetes Instrumentation Guidelines, identifying containers using pod, namespace, and container prometheus labels.The kubelet provides a gRPC service to enable discovery of in-use devices, and to provide metadata for these devices:// PodResourcesLister is a service provided by the kubelet that provides information about the // node resources consumed by pods and containers on the node service PodResourcesLister { rpc List(ListPodResourcesRequest) returns (ListPodResourcesResponse) {} rpc GetAllocatableResources(AllocatableResourcesRequest) returns (AllocatableResourcesResponse) {} rpc Get(GetPodResourcesRequest) returns (GetPodResourcesResponse) {} } List gRPC endpointThe List endpoint provides information on resources of running pods, with details such as the id of exclusively allocated CPUs, device id as it was reported by device plugins and id of the NUMA node where these devices are allocated. Also, for NUMA-based machines, it contains the information about memory and hugepages reserved for a container.Starting from Kubernetes v1.27, the List endpoint can provide information on resources of running pods allocated in ResourceClaims by the DynamicResourceAllocation API. Starting from Kubernetes v1.34, this feature is enabled by default. To disable, kubelet must be started with the following flags:--feature-gates=KubeletPodResourcesDynamicResources=false // ListPodResourcesResponse is the response returned by List function message ListPodResourcesResponse { repeated PodResources pod_resources = 1; } // PodResources contains information about the node resources assigned to a pod message PodResources { string name = 1; string namespace = 2; repeated ContainerResources containers = 3; } // ContainerResources contains information about the resources assigned to a container message ContainerResources { string name = 1; repeated ContainerDevices devices = 2; repeated int64 cpu_ids = 3; repeated ContainerMemory memory = 4; repeated DynamicResource dynamic_resources = 5; } // ContainerMemory contains information about memory and hugepages assigned to a container message ContainerMemory { string memory_type = 1; uint64 size = 2; TopologyInfo topology = 3; } // Topology describes hardware topology of the resource message TopologyInfo { repeated NUMANode nodes = 1; } // NUMA representation of NUMA node message NUMANode { int64 ID = 1; } // ContainerDevices contains information about the devices assigned to a container message ContainerDevices { string resource_name = 1; repeated string device_ids = 2; TopologyInfo topology = 3; } // DynamicResource contains information about the devices assigned to a container by Dynamic Resource Allocation message DynamicResource { string class_name = 1; string claim_name = 2; string claim_namespace = 3; repeated ClaimResource claim_resources = 4; } // ClaimResource contains per-plugin resource information message ClaimResource { repeated CDIDevice cdi_devices = 1 [(gogoproto.customname) = "CDIDevices"]; } // CDIDevice specifies a CDI device information message CDIDevice { // Fully qualified CDI device name // for example: vendor.com/gpu=gpudevice1 // see more details in the CDI specification: // https://github.com/container-orchestrated-devices/container-device-interface/blob/main/SPEC.md string name = 1; } Note:cpu_ids in the ContainerResources in the List endpoint correspond to exclusive CPUs allocated to a particular container. If the goal is to evaluate CPUs that belong to the shared pool, the List endpoint needs to be used in conjunction with the GetAllocatableResources endpoint as explained below:Call GetAllocatableResources to get a list of all the allocatable CPUsCall GetCpuIds on all ContainerResources in the systemSubtract out all of the CPUs from the GetCpuIds calls from the GetAllocatableResources callGetAllocatableResources gRPC endpointFEATURE STATE: Kubernetes v1.28 [stable]GetAllocatableResources provides information on resources initially available on the worker node. It provides more information than kubelet exports to APIServer.Note:GetAllocatableResources should only be used to evaluate allocatable resources on a node. If the goal is to evaluate free/unallocated resources it should be used in conjunction with the List() endpoint. The result obtained by GetAllocatableResources would remain the same unless the underlying resources exposed to kubelet change. This happens rarely but when it does (for example: hotplug/hotunplug, device health changes), client is expected to call GetAlloctableResources endpoint.However, calling GetAllocatableResources endpoint is not sufficient in case of cpu and/or memory update and Kubelet needs to be restarted to reflect the correct resource capacity and allocatable.// AllocatableResourcesResponses contains information about all the devices known by the kubelet message AllocatableResourcesResponse { repeated ContainerDevices devices = 1; repeated int64 cpu_ids = 2; repeated ContainerMemory memory = 3; } ContainerDevices do expose the topology information declaring to which NUMA cells the device is affine. The NUMA cells are identified using a opaque integer ID, which value is consistent to what device plugins report when they register themselves to the kubelet.The gRPC service is served over a unix socket at /var/lib/kubelet/pod-resources/kubelet.sock. Monitoring agents for device plugin resources can be deployed as a daemon, or as a DaemonSet. The canonical directory /var/lib/kubelet/pod-resources requires privileged access, so monitoring agents must run in a privileged security context. If a device monitoring agent is running as a DaemonSet, /var/lib/kubelet/pod-resources must be mounted as a Volume in the device monitoring agent's PodSpec.Note:When accessing the /var/lib/kubelet/pod-resources/kubelet.sock from DaemonSet or any other app deployed as a container on the host, which is mounting socket as a volume, it is a good practice to mount directory /var/lib/kubelet/pod-resources/ instead of the /var/lib/kubelet/pod-resources/kubelet.sock. This will ensure that after kubelet restart, container will be able to re-connect to this socket.Container mounts are managed by inode referencing the socket or directory, depending on what was mounted. When kubelet restarts, socket is deleted and a new socket is created, while directory stays untouched. So the original inode for the socket become unusable. Inode to directory will continue working.Get gRPC endpointFEATURE STATE: Kubernetes v1.34 [beta]The Get endpoint provides information on resources of a running Pod. It exposes information similar to those described in the List endpoint. The Get endpoint requires PodName and PodNamespace of the running Pod.// GetPodResourcesRequest contains information about the pod message GetPodResourcesRequest { string pod_name = 1; string pod_namespace = 2; } To disable this feature, you must start your kubelet services with the following flag:--feature-gates=KubeletPodResourcesGet=false The Get endpoint can provide Pod information related to dynamic resources allocated by the dynamic resource allocation API. Starting from Kubernetes v1.34, this feature is enabled by default. To disable, kubelet must be started with the following flags:--feature-gates=KubeletPodResourcesDynamicResources=false Device plugin integration with the Topology ManagerFEATURE STATE: Kubernetes v1.27 [stable]The Topology Manager is a Kubelet component that allows resources to be co-ordinated in a Topology aligned manner. In order to do this, the Device Plugin API was extended to include a TopologyInfo struct.message TopologyInfo { repeated NUMANode nodes = 1; } message NUMANode { int64 ID = 1; } Device Plugins that wish to leverage the Topology Manager can send back a populated TopologyInfo struct as part of the device registration, along with the device IDs and the health of the device. The device manager will then use this information to consult with the Topology Manager and make resource assignment decisions.TopologyInfo supports setting a nodes field to either nil or a list of NUMA nodes. This allows the Device Plugin to advertise a device that spans multiple NUMA nodes.Setting TopologyInfo to nil or providing an empty list of NUMA nodes for a given device indicates that the Device Plugin does not have a NUMA affinity preference for that device.An example TopologyInfo struct populated for a device by a Device Plugin:pluginapi.Device{ID: "25102017", Health: pluginapi.Healthy, Topology:&pluginapi.TopologyInfo{Nodes: []*pluginapi.NUMANode{&pluginapi.NUMANode{ID: 0,},}}} Device plugin examplesNote: This section links to third party projects that provide functionality required by Kubernetes. The Kubernetes project authors aren't responsible for these projects, which are listed alphabetically. To add a project to this list, read the content guide before submitting a change. More information.Here are some examples of device plugin implementations:Akri, which lets you easily expose heterogeneous leaf devices (such as IP cameras and USB devices).The AMD GPU device pluginThe generic device plugin for generic Linux devices and USB devicesThe HAMi for heterogeneous AI computing virtualization middleware (for example, NVIDIA, Cambricon, Hygon, Iluvatar, MThreads, Ascend, Metax)The Intel device plugins for Intel GPU, FPGA, QAT, VPU, SGX, DSA, DLB and IAA devicesThe KubeVirt device plugins for hardware-assisted virtualizationThe NVIDIA GPU device plugin, NVIDIA's official device plugin to expose NVIDIA GPUs and monitor GPU healthThe NVIDIA GPU device plugin for Container-Optimized OSThe RDMA device pluginThe SocketCAN device pluginThe Solarflare device pluginThe SR-IOV Network device pluginThe Xilinx FPGA device plugins for Xilinx FPGA devicesWhat's nextLearn about scheduling GPU resources using device pluginsLearn about advertising extended resources on a nodeLearn about the Topology ManagerRead about using hardware acceleration for TLS ingress with KubernetesRead more about Extended Resource allocation by DRAItems on this page refer to third party products or projects that provide functionality required by Kubernetes. The Kubernetes project authors aren't responsible for those third-party products or projects. See the CNCF website guidelines for more details.You should read the content guide before proposing a change that adds an extra third-party link.

Device Plugins


Assign CPU Resources to Containers and PodsThis page shows how to assign a CPU request and a CPU limit to a container. Containers cannot use more CPU than the configured limit. Provided the system has CPU time free, a container is guaranteed to be allocated as much CPU as it requests.Before you beginYou need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. It is recommended to run this tutorial on a cluster with at least two nodes that are not acting as control plane hosts. If you do not already have a cluster, you can create one by using minikube or you can use one of these Kubernetes playgrounds:iximiuz LabsKillercodaKodeKloudPlay with KubernetesTo check the version, enter kubectl version.Your cluster must have at least 1 CPU available for use to run the task examples.A few of the steps on this page require you to run the metrics-server service in your cluster. If you have the metrics-server running, you can skip those steps.If you are running Minikube, run the following command to enable metrics-server:minikube addons enable metrics-server To see whether metrics-server (or another provider of the resource metrics API, metrics.k8s.io) is running, type the following command:kubectl get apiservices If the resource metrics API is available, the output will include a reference to metrics.k8s.io.NAME v1beta1.metrics.k8s.io Create a namespaceCreate a Namespace so that the resources you create in this exercise are isolated from the rest of your cluster.kubectl create namespace cpu-example Specify a CPU request and a CPU limitTo specify a CPU request for a container, include the resources:requests field in the Container resource manifest. To specify a CPU limit, include resources:limits.In this exercise, you create a Pod that has one container. The container has a request of 0.5 CPU and a limit of 1 CPU. Here is the configuration file for the Pod:pods/resource/cpu-request-limit.yaml apiVersion: v1 kind: Pod metadata: name: cpu-demo namespace: cpu-example spec: containers: - name: cpu-demo-ctr image: vish/stress resources: limits: cpu: "1" requests: cpu: "0.5" args: - -cpus - "2" The args section of the configuration file provides arguments for the container when it starts. The -cpus "2" argument tells the Container to attempt to use 2 CPUs.Create the Pod:kubectl apply -f https://k8s.io/examples/pods/resource/cpu-request-limit.yaml --namespace=cpu-example Verify that the Pod is running:kubectl get pod cpu-demo --namespace=cpu-example View detailed information about the Pod:kubectl get pod cpu-demo --output=yaml --namespace=cpu-example The output shows that the one container in the Pod has a CPU request of 500 milliCPU and a CPU limit of 1 CPU.resources: limits: cpu: "1" requests: cpu: 500m Use kubectl top to fetch the metrics for the Pod:kubectl top pod cpu-demo --namespace=cpu-example This example output shows that the Pod is using 974 milliCPU, which is slightly less than the limit of 1 CPU specified in the Pod configuration.NAME CPU(cores) MEMORY(bytes) cpu-demo 974m <something> Recall that by setting -cpu "2", you configured the Container to attempt to use 2 CPUs, but the Container is only being allowed to use about 1 CPU. The container's CPU use is being throttled, because the container is attempting to use more CPU resources than its limit.Note:Another possible explanation for the CPU use being below 1.0 is that the Node might not have enough CPU resources available. Recall that the prerequisites for this exercise require your cluster to have at least 1 CPU available for use. If your Container runs on a Node that has only 1 CPU, the Container cannot use more than 1 CPU regardless of the CPU limit specified for the Container.CPU unitsThe CPU resource is measured in CPU units. One CPU, in Kubernetes, is equivalent to:1 AWS vCPU1 GCP Core1 Azure vCore1 Hyperthread on a bare-metal Intel processor with HyperthreadingFractional values are allowed. A Container that requests 0.5 CPU is guaranteed half as much CPU as a Container that requests 1 CPU. You can use the suffix m to mean milli. For example 100m CPU, 100 milliCPU, and 0.1 CPU are all the same. Precision finer than 1m is not allowed.CPU is always requested as an absolute quantity, never as a relative quantity; 0.1 is the same amount of CPU on a single-core, dual-core, or 48-core machine.Delete your Pod:kubectl delete pod cpu-demo --namespace=cpu-example Specify a CPU request that is too big for your NodesCPU requests and limits are associated with Containers, but it is useful to think of a Pod as having a CPU request and limit. The CPU request for a Pod is the sum of the CPU requests for all the Containers in the Pod. Likewise, the CPU limit for a Pod is the sum of the CPU limits for all the Containers in the Pod.Pod scheduling is based on requests. A Pod is scheduled to run on a Node only if the Node has enough CPU resources available to satisfy the Pod CPU request.In this exercise, you create a Pod that has a CPU request so big that it exceeds the capacity of any Node in your cluster. Here is the configuration file for a Pod that has one Container. The Container requests 100 CPU, which is likely to exceed the capacity of any Node in your cluster.pods/resource/cpu-request-limit-2.yaml apiVersion: v1 kind: Pod metadata: name: cpu-demo-2 namespace: cpu-example spec: containers: - name: cpu-demo-ctr-2 image: vish/stress resources: limits: cpu: "100" requests: cpu: "100" args: - -cpus - "2" Create the Pod:kubectl apply -f https://k8s.io/examples/pods/resource/cpu-request-limit-2.yaml --namespace=cpu-example View the Pod status:kubectl get pod cpu-demo-2 --namespace=cpu-example The output shows that the Pod status is Pending. That is, the Pod has not been scheduled to run on any Node, and it will remain in the Pending state indefinitely:NAME READY STATUS RESTARTS AGE cpu-demo-2 0/1 Pending 0 7m View detailed information about the Pod, including events:kubectl describe pod cpu-demo-2 --namespace=cpu-example The output shows that the Container cannot be scheduled because of insufficient CPU resources on the Nodes:Events: Reason Message ------ ------- FailedScheduling No nodes are available that match all of the following predicates:: Insufficient cpu (3). Delete your Pod:kubectl delete pod cpu-demo-2 --namespace=cpu-example If you do not specify a CPU limitIf you do not specify a CPU limit for a Container, then one of these situations applies:The Container has no upper bound on the CPU resources it can use. The Container could use all of the CPU resources available on the Node where it is running.The Container is running in a namespace that has a default CPU limit, and the Container is automatically assigned the default limit. Cluster administrators can use a LimitRange to specify a default value for the CPU limit.If you specify a CPU limit but do not specify a CPU requestIf you specify a CPU limit for a Container but do not specify a CPU request, Kubernetes automatically assigns a CPU request that matches the limit. Similarly, if a Container specifies its own memory limit, but does not specify a memory request, Kubernetes automatically assigns a memory request that matches the limit.Motivation for CPU requests and limitsBy configuring the CPU requests and limits of the Containers that run in your cluster, you can make efficient use of the CPU resources available on your cluster Nodes. By keeping a Pod CPU request low, you give the Pod a good chance of being scheduled. By having a CPU limit that is greater than the CPU request, you accomplish two things:The Pod can have bursts of activity where it makes use of CPU resources that happen to be available.The amount of CPU resources a Pod can use during a burst is limited to some reasonable amount.Clean upDelete your namespace:kubectl delete namespace cpu-example What's nextFor app developersAssign Memory Resources to Containers and PodsAssign Pod-level CPU and memory resourcesConfigure Quality of Service for PodsResize CPU and Memory Resources assigned to ContainersFor cluster administratorsConfigure Default Memory Requests and Limits for a NamespaceConfigure Default CPU Requests and Limits for a NamespaceConfigure Minimum and Maximum Memory Constraints for a NamespaceConfigure Minimum and Maximum CPU Constraints for a NamespaceConfigure Memory and CPU Quotas for a NamespaceConfigure a Pod Quota for a NamespaceConfigure Quotas for API ObjectsResize CPU and Memory Resources assigned to Containers


Autoscaling WorkloadsWith autoscaling, you can automatically update your workloads in one way or another. This allows your cluster to react to changes in resource demand more elastically and efficiently.In Kubernetes, you can scale a workload depending on the current demand of resources. This allows your cluster to react to changes in resource demand more elastically and efficiently.When you scale a workload, you can either increase or decrease the number of replicas managed by the workload, or adjust the resources available to the replicas in-place.The first approach is referred to as horizontal scaling, while the second is referred to as vertical scaling.There are manual and automatic ways to scale your workloads, depending on your use case.Scaling workloads manuallyKubernetes supports manual scaling of workloads. Horizontal scaling can be done using the kubectl CLI. For vertical scaling, you need to patch the resource definition of your workload.See below for examples of both strategies.Horizontal scaling: Running multiple instances of your appVertical scaling: Resizing CPU and memory resources assigned to containersScaling workloads automaticallyKubernetes also supports automatic scaling of workloads, which is the focus of this page.The concept of Autoscaling in Kubernetes refers to the ability to automatically update an object that manages a set of Pods (for example a Deployment).Scaling workloads horizontallyIn Kubernetes, you can automatically scale a workload horizontally using a HorizontalPodAutoscaler (HPA).It is implemented as a Kubernetes API resource and a controller and periodically adjusts the number of replicas in a workload to match observed resource utilization such as CPU or memory usage.There is a walkthrough tutorial of configuring a HorizontalPodAutoscaler for a Deployment.Scaling workloads verticallyFEATURE STATE: Kubernetes v1.25 [stable]You can automatically scale a workload vertically using a VerticalPodAutoscaler (VPA). Unlike the HPA, the VPA doesn't come with Kubernetes by default, but is a separate project that can be found on GitHub.Once installed, it allows you to create CustomResourceDefinitions (CRDs) for your workloads which define how and when to scale the resources of the managed replicas.Note:You will need to have the Metrics Server installed to your cluster for the VPA to work.At the moment, the VPA can operate in four different modes:Different modes of the VPAModeDescriptionAutoCurrently Recreate. This might change to in-place updates in the future.RecreateThe VPA assigns resource requests on pod creation as well as updates them on existing pods by evicting them when the requested resources differ significantly from the new recommendationInitialThe VPA only assigns resource requests on pod creation and never changes them later.OffThe VPA does not automatically change the resource requirements of the pods. The recommendations are calculated and can be inspected in the VPA object.In-place pod vertical scalingFEATURE STATE: Kubernetes v1.33 [beta] (enabled by default: true)As of Kubernetes 1.34, VPA does not support resizing pods in-place, but this integration is being worked on. For manually resizing pods in-place, see Resize Container Resources In-Place.Autoscaling based on cluster sizeFor workloads that need to be scaled based on the size of the cluster (for example cluster-dns or other system components), you can use the Cluster Proportional Autoscaler. Just like the VPA, it is not part of the Kubernetes core, but hosted as its own project on GitHub.The Cluster Proportional Autoscaler watches the number of schedulable nodes and cores and scales the number of replicas of the target workload accordingly.If the number of replicas should stay the same, you can scale your workloads vertically according to the cluster size using the Cluster Proportional Vertical Autoscaler. The project is currently in beta and can be found on GitHub.While the Cluster Proportional Autoscaler scales the number of replicas of a workload, the Cluster Proportional Vertical Autoscaler adjusts the resource requests for a workload (for example a Deployment or DaemonSet) based on the number of nodes and/or cores in the cluster.Event driven AutoscalingIt is also possible to scale workloads based on events, for example using the Kubernetes Event Driven Autoscaler (KEDA).KEDA is a CNCF-graduated project enabling you to scale your workloads based on the number of events to be processed, for example the amount of messages in a queue. There exists a wide range of adapters for different event sources to choose from.Autoscaling based on schedulesAnother strategy for scaling your workloads is to schedule the scaling operations, for example in order to reduce resource consumption during off-peak hours.Similar to event driven autoscaling, such behavior can be achieved using KEDA in conjunction with its Cron scaler. The Cron scaler allows you to define schedules (and time zones) for scaling your workloads in or out.Scaling cluster infrastructureIf scaling workloads isn't enough to meet your needs, you can also scale your cluster infrastructure itself.Scaling the cluster infrastructure normally means adding or removing nodes. Read Node autoscaling for more information.What's nextLearn more about scaling horizontallyScale a StatefulSetHorizontalPodAutoscaler WalkthroughResize Container Resources In-PlaceAutoscale the DNS Service in a ClusterLearn about Node autoscaling

Autoscaling Workloads


Documentation Style GuideThis page gives writing style guidelines for the Kubernetes documentation. These are guidelines, not rules. Use your best judgment, and feel free to propose changes to this document in a pull request.For additional information on creating new content for the Kubernetes documentation, read the Documentation Content Guide.Changes to the style guide are made by SIG Docs as a group. To propose a change or addition, add it to the agenda for an upcoming SIG Docs meeting, and attend the meeting to participate in the discussion.Note:Kubernetes documentation uses Goldmark Markdown Renderer with some adjustments along with a few Hugo Shortcodes to support glossary entries, tabs, and representing feature state.LanguageKubernetes documentation has been translated into multiple languages (see Localization READMEs).The way of localizing the docs for a different language is described in Localizing Kubernetes Documentation.The English-language documentation uses U.S. English spelling and grammar.Documentation formatting standardsUse upper camel case for API objectsWhen you refer specifically to interacting with an API object, use UpperCamelCase, also known as Pascal case. You may see different capitalization, such as "configMap", in the API Reference. When writing general documentation, it's better to use upper camel case, calling it "ConfigMap" instead.When you are generally discussing an API object, use sentence-style capitalization.The following examples focus on capitalization. For more information about formatting API object names, review the related guidance on Code Style.Do and Don't - Use Pascal case for API objectsDoDon'tThe HorizontalPodAutoscaler resource is responsible for ...The Horizontal pod autoscaler is responsible for ...A PodList object is a list of pods.A Pod List object is a list of pods.The Volume object contains a hostPath field.The volume object contains a hostPath field.Every ConfigMap object is part of a namespace.Every configMap object is part of a namespace.For managing confidential data, consider using the Secret API.For managing confidential data, consider using the secret API.Use angle brackets for placeholdersUse angle brackets for placeholders. Tell the reader what a placeholder represents, for example:Display information about a pod:kubectl describe pod <pod-name> -n <namespace> If the namespace of the pod is default, you can omit the '-n' parameter.Use bold for user interface elementsDo and Don't - Bold interface elementsDoDon'tClick Fork.Click "Fork".Select Other.Select "Other".Use italics to define or introduce new termsDo and Don't - Use italics for new termsDoDon'tA cluster is a set of nodes ...A "cluster" is a set of nodes ...These components form the control plane.These components form the control plane.Use code style for filenames, directories, and pathsDo and Don't - Use code style for filenames, directories, and pathsDoDon'tOpen the envars.yaml file.Open the envars.yaml file.Go to the /docs/tutorials directory.Go to the /docs/tutorials directory.Open the /_data/concepts.yaml file.Open the /_data/concepts.yaml file.Use the international standard for punctuation inside quotesDo and Don't - Use the international standard for punctuation inside quotesDoDon'tevents are recorded with an associated "stage".events are recorded with an associated "stage."The copy is called a "fork".The copy is called a "fork."Inline code formattingUse code style for inline code, commandsFor inline code in an HTML document, use the <code> tag. In a Markdown document, use the backtick (`). However, API kinds such as StatefulSet or ConfigMap are written verbatim (no backticks); this allows using possessive apostrophes.Do and Don't - Use code style for inline code, commands, and API objectsDoDon'tThe kubectl run command creates a Pod.The "kubectl run" command creates a Pod.The kubelet on each node acquires a Lease…The kubelet on each node acquires a Lease…A PersistentVolume represents durable storage…A PersistentVolume represents durable storage…The CustomResourceDefinition's .spec.group field…The CustomResourceDefinition.spec.group field…For declarative management, use kubectl apply.For declarative management, use "kubectl apply".Enclose code samples with triple backticks. (```)Enclose code samples with any other syntax.Use single backticks to enclose inline code. For example, var example = true.Use two asterisks (**) or an underscore (_) to enclose inline code. For example, var example = true.Use triple backticks before and after a multi-line block of code for fenced code blocks.Use multi-line blocks of code to create diagrams, flowcharts, or other illustrations.Use meaningful variable names that have a context.Use variable names such as 'foo','bar', and 'baz' that are not meaningful and lack context.Remove trailing spaces in the code.Add trailing spaces in the code, where these are important, because the screen reader will read out the spaces as well.Note:The website supports syntax highlighting for code samples, but specifying a language is optional. Syntax highlighting in the code block should conform to the contrast guidelines.Use code style for object field names and namespacesDo and Don't - Use code style for object field namesDoDon'tSet the value of the replicas field in the configuration file.Set the value of the "replicas" field in the configuration file.The value of the exec field is an ExecAction object.The value of the "exec" field is an ExecAction object.Run the process as a DaemonSet in the kube-system namespace.Run the process as a DaemonSet in the kube-system namespace.Use code style for Kubernetes command tool and component namesDo and Don't - Use code style for Kubernetes command tool and component namesDoDon'tThe kubelet preserves node stability.The kubelet preserves node stability.The kubectl handles locating and authenticating to the API server.The kubectl handles locating and authenticating to the apiserver.Run the process with the certificate, kube-apiserver --client-ca-file=FILENAME.Run the process with the certificate, kube-apiserver --client-ca-file=FILENAME.Starting a sentence with a component tool or component nameDo and Don't - Starting a sentence with a component tool or component nameDoDon'tThe kubeadm tool bootstraps and provisions machines in a cluster.kubeadm tool bootstraps and provisions machines in a cluster.The kube-scheduler is the default scheduler for Kubernetes.kube-scheduler is the default scheduler for Kubernetes.Use a general descriptor over a component nameDo and Don't - Use a general descriptor over a component nameDoDon'tThe Kubernetes API server offers an OpenAPI spec.The apiserver offers an OpenAPI spec.Aggregated APIs are subordinate API servers.Aggregated APIs are subordinate APIServers.Use normal style for string and integer field valuesFor field values of type string or integer, use normal style without quotation marks.Do and Don't - Use normal style for string and integer field valuesDoDon'tSet the value of imagePullPolicy to Always.Set the value of imagePullPolicy to "Always".Set the value of image to nginx:1.16.Set the value of image to nginx:1.16.Set the value of the replicas field to 2.Set the value of the replicas field to 2.However, consider quoting values where there is a risk that readers might confuse the value with an API kind.Referring to Kubernetes API resourcesThis section talks about how we reference API resources in the documentation.Clarification about "resource"Kubernetes uses the word resource to refer to API resources. For example, the URL path /apis/apps/v1/namespaces/default/deployments/my-app represents a Deployment named "my-app" in the "default" namespace. In HTTP jargon, namespace is a resource - the same way that all web URLs identify a resource.Kubernetes documentation also uses "resource" to talk about CPU and memory requests and limits. It's very often a good idea to refer to API resources as "API resources"; that helps to avoid confusion with CPU and memory resources, or with other kinds of resource.If you are using the lowercase plural form of a resource name, such as deployments or configmaps, provide extra written context to help readers understand what you mean. If you are using the term in a context where the UpperCamelCase name could work too, and there is a risk of ambiguity, consider using the API kind in UpperCamelCase.When to use Kubernetes API terminologiesThe different Kubernetes API terminologies are:API kinds: the name used in the API URL (such as pods, namespaces). API kinds are sometimes also called resource types.API resource: a single instance of an API kind (such as pod, secret).Object: a resource that serves as a "record of intent". An object is a desired state for a specific part of your cluster, which the Kubernetes control plane tries to maintain. All objects in the Kubernetes API are also resources.For clarity, you can add "resource" or "object" when referring to an API resource in Kubernetes documentation. An example: write "a Secret object" instead of "a Secret". If it is clear just from the capitalization, you don't need to add the extra word.Consider rephrasing when that change helps avoid misunderstandings. A common situation is when you want to start a sentence with an API kind, such as “Secret”; because English and other languages capitalize at the start of sentences, readers cannot tell whether you mean the API kind or the general concept. Rewording can help.API resource namesAlways format API resource names using UpperCamelCase, also known as PascalCase. Do not write API kinds with code formatting.Don't split an API object name into separate words. For example, use PodTemplateList, not Pod Template List.For more information about PascalCase and code formatting, review the related guidance on Use upper camel case for API objects and Use code style for inline code, commands, and API objects.For more information about Kubernetes API terminologies, review the related guidance on Kubernetes API terminology.Code snippet formattingDon't include the command promptDo and Don't - Don't include the command promptDoDon'tkubectl get pods$ kubectl get podsSeparate commands from outputVerify that the pod is running on your chosen node:kubectl get pods --output=wide The output is similar to this:NAME READY STATUS RESTARTS AGE IP NODE nginx 1/1 Running 0 13s 10.200.0.4 worker0 Versioning Kubernetes examplesCode examples and configuration examples that include version information should be consistent with the accompanying text.If the information is version specific, the Kubernetes version needs to be defined in the prerequisites section of the Task template or the Tutorial template. Once the page is saved, the prerequisites section is shown as Before you begin.To specify the Kubernetes version for a task or tutorial page, include min-kubernetes-server-version in the front matter of the page.If the example YAML is in a standalone file, find and review the topics that include it as a reference. Verify that any topics using the standalone YAML have the appropriate version information defined. If a stand-alone YAML file is not referenced from any topics, consider deleting it instead of updating it.For example, if you are writing a tutorial that is relevant to Kubernetes version 1.8, the front-matter of your markdown file should look something like:--- title: <your tutorial title here> min-kubernetes-server-version: v1.8 --- In code and configuration examples, do not include comments about alternative versions. Be careful to not include incorrect statements in your examples as comments, such as:apiVersion: v1 # earlier versions use... kind: Pod ... Formulae and equationsYou can use the Docsy support for diagrams and formulae.For example: \\(\frac{7}{9} \sqrt{K^8 s}\\), which renders as \(\frac{7}{9} \sqrt{K^8 s}\).Prefer inline formulae where reasonable, but you can use a math block if that's likely to help readers.Read the Docsy guide to find out what you need to change in your page to activate support; if you have problems, add math: true to the page front matter (you can do this even if you think the automatic activation should be enough).Kubernetes.io word listA list of Kubernetes-specific terms and words to be used consistently across the site.Kubernetes.io word listTermUsageKubernetesKubernetes should always be capitalized.DockerDocker should always be capitalized.SIG DocsSIG Docs rather than SIG-DOCS or other variations.On-premisesOn-premises or On-prem rather than On-premise or other variations.cloud nativeCloud native or cloud native as appropriate for sentence structure rather than cloud-native or Cloud Native.open sourceOpen source or open source as appropriate for sentence structure rather than open-source or Open Source.ShortcodesHugo Shortcodes help create different rhetorical appeal levels. Our documentation supports three different shortcodes in this category: Note {{< note >}}, Caution {{< caution >}}, and Warning {{< warning >}}.Surround the text with an opening and closing shortcode.Use the following syntax to apply a style:{{< note >}} No need to include a prefix; the shortcode automatically provides one. (Note:, Caution:, etc.) {{< /note >}} The output is:Note:The prefix you choose is the same text for the tag.NoteUse {{< note >}} to highlight a tip or a piece of information that may be helpful to know.For example:{{< note >}} You can _still_ use Markdown inside these callouts. {{< /note >}} The output is:Note:You can still use Markdown inside these callouts.You can use a {{< note >}} in a list:1. Use the note shortcode in a list 1. A second item with an embedded note {{< note >}} Warning, Caution, and Note shortcodes, embedded in lists, need to be indented four spaces. See [Common Shortcode Issues](#common-shortcode-issues). {{< /note >}} 1. A third item in a list 1. A fourth item in a list The output is:Use the note shortcode in a listA second item with an embedded noteNote:Warning, Caution, and Note shortcodes, embedded in lists, need to be indented four spaces. See [Common Shortcode Issues](#common-shortcode-issues). A third item in a listA fourth item in a listCautionUse {{< caution >}} to call attention to an important piece of information to avoid pitfalls.For example:{{< caution >}} The callout style only applies to the line directly above the tag. {{< /caution >}} The output is:Caution:The callout style only applies to the line directly above the tag.WarningUse {{< warning >}} to indicate danger or a piece of information that is crucial to follow.For example:{{< warning >}} Beware. {{< /warning >}} The output is:Warning:Beware.Common Shortcode IssuesOrdered ListsShortcodes will interrupt numbered lists unless you indent four spaces before the notice and the tag.For example:1. Preheat oven to 350˚F 1. Prepare the batter, and pour into springform pan. {{< note >}}Grease the pan for best results.{{< /note >}} 1. Bake for 20-25 minutes or until set. The output is:Preheat oven to 350˚FPrepare the batter, and pour into springform pan.Note:Grease the pan for best results.Bake for 20-25 minutes or until set.Include StatementsShortcodes inside include statements will break the build. You must insert them in the parent document, before and after you call the include. For example:{{< note >}} {{< include "task-tutorial-prereqs.md" >}} {{< /note >}} Markdown elementsLine breaksUse a single newline to separate block-level content like headings, lists, images, code blocks, and others. The exception is second-level headings, where it should be two newlines. Second-level headings follow the first-level (or the title) without any preceding paragraphs or texts. A two line spacing helps visualize the overall structure of content in a code editor better.Manually wrap paragraphs in the Markdown source when appropriate. Since the git tool and the GitHub website generate file diffs on a line-by-line basis, manually wrapping long lines helps the reviewers to easily find out the changes made in a PR and provide

Documentation Style Guide


Pod LifecycleThis page describes the lifecycle of a Pod. Pods follow a defined lifecycle, starting in the Pending phase, moving through Running if at least one of its primary containers starts OK, and then through either the Succeeded or Failed phases depending on whether any container in the Pod terminated in failure.Like individual application containers, Pods are considered to be relatively ephemeral (rather than durable) entities. Pods are created, assigned a unique ID (UID), and scheduled to run on nodes where they remain until termination (according to restart policy) or deletion. If a Node dies, the Pods running on (or scheduled to run on) that node are marked for deletion. The control plane marks the Pods for removal after a timeout period.Pod lifetimeWhilst a Pod is running, the kubelet is able to restart containers to handle some kind of faults. Within a Pod, Kubernetes tracks different container states and determines what action to take to make the Pod healthy again.In the Kubernetes API, Pods have both a specification and an actual status. The status for a Pod object consists of a set of Pod conditions. You can also inject custom readiness information into the condition data for a Pod, if that is useful to your application.Pods are only scheduled once in their lifetime; assigning a Pod to a specific node is called binding, and the process of selecting which node to use is called scheduling. Once a Pod has been scheduled and is bound to a node, Kubernetes tries to run that Pod on the node. The Pod runs on that node until it stops, or until the Pod is terminated; if Kubernetes isn't able to start the Pod on the selected node (for example, if the node crashes before the Pod starts), then that particular Pod never starts.You can use Pod Scheduling Readiness to delay scheduling for a Pod until all its scheduling gates are removed. For example, you might want to define a set of Pods but only trigger scheduling once all the Pods have been created.Pods and fault recoveryIf one of the containers in the Pod fails, then Kubernetes may try to restart that specific container. Read How Pods handle problems with containers to learn more.Pods can however fail in a way that the cluster cannot recover from, and in that case Kubernetes does not attempt to heal the Pod further; instead, Kubernetes deletes the Pod and relies on other components to provide automatic healing.If a Pod is scheduled to a node and that node then fails, the Pod is treated as unhealthy and Kubernetes eventually deletes the Pod. A Pod won't survive an eviction due to a lack of resources or Node maintenance.Kubernetes uses a higher-level abstraction, called a controller, that handles the work of managing the relatively disposable Pod instances.A given Pod (as defined by a UID) is never "rescheduled" to a different node; instead, that Pod can be replaced by a new, near-identical Pod. If you make a replacement Pod, it can even have same name (as in .metadata.name) that the old Pod had, but the replacement would have a different .metadata.uid from the old Pod.Kubernetes does not guarantee that a replacement for an existing Pod would be scheduled to the same node as the old Pod that was being replaced.Associated lifetimesWhen something is said to have the same lifetime as a Pod, such as a volume, that means that the thing exists as long as that specific Pod (with that exact UID) exists. If that Pod is deleted for any reason, and even if an identical replacement is created, the related thing (a volume, in this example) is also destroyed and created anew.Figure 1.A multi-container Pod that contains a file puller sidecar and a web server. The Pod uses an ephemeral emptyDir volume for shared storage between the containers.Pod phaseA Pod's status field is a PodStatus object, which has a phase field.The phase of a Pod is a simple, high-level summary of where the Pod is in its lifecycle. The phase is not intended to be a comprehensive rollup of observations of container or Pod state, nor is it intended to be a comprehensive state machine.The number and meanings of Pod phase values are tightly guarded. Other than what is documented here, nothing should be assumed about Pods that have a given phase value.Here are the possible values for phase:ValueDescriptionPendingThe Pod has been accepted by the Kubernetes cluster, but one or more of the containers has not been set up and made ready to run. This includes time a Pod spends waiting to be scheduled as well as the time spent downloading container images over the network.RunningThe Pod has been bound to a node, and all of the containers have been created. At least one container is still running, or is in the process of starting or restarting.SucceededAll containers in the Pod have terminated in success, and will not be restarted.FailedAll containers in the Pod have terminated, and at least one container has terminated in failure. That is, the container either exited with non-zero status or was terminated by the system, and is not set for automatic restarting.UnknownFor some reason the state of the Pod could not be obtained. This phase typically occurs due to an error in communicating with the node where the Pod should be running.Note:When a pod is failing to start repeatedly, CrashLoopBackOff may appear in the Status field of some kubectl commands. Similarly, when a pod is being deleted, Terminating may appear in the Status field of some kubectl commands.Make sure not to confuse Status, a kubectl display field for user intuition, with the pod's phase. Pod phase is an explicit part of the Kubernetes data model and of the Pod API. NAMESPACE NAME READY STATUS RESTARTS AGE alessandras-namespace alessandras-pod 0/1 CrashLoopBackOff 200 2d9h A Pod is granted a term to terminate gracefully, which defaults to 30 seconds. You can use the flag --force to terminate a Pod by force.Since Kubernetes 1.27, the kubelet transitions deleted Pods, except for static Pods and force-deleted Pods without a finalizer, to a terminal phase (Failed or Succeeded depending on the exit statuses of the pod containers) before their deletion from the API server.If a node dies or is disconnected from the rest of the cluster, Kubernetes applies a policy for setting the phase of all Pods on the lost node to Failed.Container statesAs well as the phase of the Pod overall, Kubernetes tracks the state of each container inside a Pod. You can use container lifecycle hooks to trigger events to run at certain points in a container's lifecycle.Once the scheduler assigns a Pod to a Node, the kubelet starts creating containers for that Pod using a container runtime. There are three possible container states: Waiting, Running, and Terminated.To check the state of a Pod's containers, you can use kubectl describe pod <name-of-pod>. The output shows the state for each container within that Pod.Each state has a specific meaning:WaitingIf a container is not in either the Running or Terminated state, it is Waiting. A container in the Waiting state is still running the operations it requires in order to complete start up: for example, pulling the container image from a container image registry, or applying Secret data. When you use kubectl to query a Pod with a container that is Waiting, you also see a Reason field to summarize why the container is in that state.RunningThe Running status indicates that a container is executing without issues. If there was a postStart hook configured, it has already executed and finished. When you use kubectl to query a Pod with a container that is Running, you also see information about when the container entered the Running state.TerminatedA container in the Terminated state began execution and then either ran to completion or failed for some reason. When you use kubectl to query a Pod with a container that is Terminated, you see a reason, an exit code, and the start and finish time for that container's period of execution.If a container has a preStop hook configured, this hook runs before the container enters the Terminated state.How Pods handle problems with containersKubernetes manages container failures within Pods using a restartPolicy defined in the Pod spec. This policy determines how Kubernetes reacts to containers exiting due to errors or other reasons, which falls in the following sequence:Initial crash: Kubernetes attempts an immediate restart based on the Pod restartPolicy.Repeated crashes: After the initial crash Kubernetes applies an exponential backoff delay for subsequent restarts, described in restartPolicy. This prevents rapid, repeated restart attempts from overloading the system.CrashLoopBackOff state: This indicates that the backoff delay mechanism is currently in effect for a given container that is in a crash loop, failing and restarting repeatedly.Backoff reset: If a container runs successfully for a certain duration (e.g., 10 minutes), Kubernetes resets the backoff delay, treating any new crash as the first one.In practice, a CrashLoopBackOff is a condition or event that might be seen as output from the kubectl command, while describing or listing Pods, when a container in the Pod fails to start properly and then continually tries and fails in a loop.In other words, when a container enters the crash loop, Kubernetes applies the exponential backoff delay mentioned in the Container restart policy. This mechanism prevents a faulty container from overwhelming the system with continuous failed start attempts.The CrashLoopBackOff can be caused by issues like the following:Application errors that cause the container to exit.Configuration errors, such as incorrect environment variables or missing configuration files.Resource constraints, where the container might not have enough memory or CPU to start properly.Health checks failing if the application doesn't start serving within the expected time.Container liveness probes or startup probes returning a Failure result as mentioned in the probes section.To investigate the root cause of a CrashLoopBackOff issue, a user can:Check logs: Use kubectl logs <name-of-pod> to check the logs of the container. This is often the most direct way to diagnose the issue causing the crashes.Inspect events: Use kubectl describe pod <name-of-pod> to see events for the Pod, which can provide hints about configuration or resource issues.Review configuration: Ensure that the Pod configuration, including environment variables and mounted volumes, is correct and that all required external resources are available.Check resource limits: Make sure that the container has enough CPU and memory allocated. Sometimes, increasing the resources in the Pod definition can resolve the issue.Debug application: There might exist bugs or misconfigurations in the application code. Running this container image locally or in a development environment can help diagnose application specific issues.Container restartsWhen a container in your Pod stops, or experiences failure, Kubernetes can restart it. A restart isn't always appropriate; for example, init containers run only once, during Pod startup.You can configure restarts as a policy that applies to all Pods, or using container-level configuration (for example: when you define a sidecar container).Container restarts and resilienceThe Kubernetes project recommends following cloud-native principles, including resilient design that accounts for unannounced or arbitrary restarts. You can achieve this either by failing the Pod and relying on automatic replacement, or you can design for container-level resilience. Either approach helps to ensure that your overall workload remains available despite partial failure.Pod-level container restart policyThe spec of a Pod has a restartPolicy field with possible values Always, OnFailure, and Never. The default value is Always.The restartPolicy for a Pod applies to app containers in the Pod and to regular init containers. Sidecar containers ignore the Pod-level restartPolicy field: in Kubernetes, a sidecar is defined as an entry inside initContainers that has its container-level restartPolicy set to Always. For init containers that exit with an error, the kubelet restarts the init container if the Pod level restartPolicy is either OnFailure or Always:Always: Automatically restarts the container after any termination.OnFailure: Only restarts the container if it exits with an error (non-zero exit status).Never: Does not automatically restart the terminated container.When the kubelet is handling container restarts according to the configured restart policy, that only applies to restarts that make replacement containers inside the same Pod and running on the same node. After containers in a Pod exit, the kubelet restarts them with an exponential backoff delay (10s, 20s, 40s, …), that is capped at 300 seconds (5 minutes). Once a container has executed for 10 minutes without any problems, the kubelet resets the restart backoff timer for that container. Sidecar containers and Pod lifecycle explains the behaviour of init containers when specify restartpolicy field on it.Individual container restart policy and rulesFEATURE STATE: Kubernetes v1.34 [alpha] (enabled by default: false)If your cluster has the feature gate ContainerRestartRules enabled, you can specify restartPolicy and restartPolicyRules on individual containers to override the Pod restart policy. Container restart policy and rules applies to app containers in the Pod and to regular init containers.A Kubernetes-native sidecar container has its container-level restartPolicy set to Always, and does not support restartPolicyRules.The container restarts will follow the same exponential backoff as pod restart policy described above. Supported container restart policies:Always: Automatically restarts the container after any termination.OnFailure: Only restarts the container if it exits with an error (non-zero exit status).Never: Does not automatically restart the terminated container.Additionally, individual containers can specify restartPolicyRules. If the restartPolicyRules field is specified, then container restartPolicy must also be specified. The restartPolicyRules define a list of rules to apply on container exit. Each rule will consist of a condition and an action. The supported condition is exitCodes, which compares the exit code of the container with a list of given values. The supported action is Restart, which means the container will be restarted. The rules will be evaluated in order. On the first match, the action will be applied. If none of the rules’ conditions matched, Kubernetes fallback to container’s configured restartPolicy.For example, a Pod with OnFailure restart policy that have a try-once container. This allows Pod to only restart certain containers:apiVersion: v1 kind: Pod metadata: name: on-failure-pod spec: restartPolicy: OnFailure containers: - name: try-once-container # This container will run only once because the restartPolicy is Never. image: docker.io/library/busybox:1.28 command: ['sh', '-c', 'echo "Only running once" && sleep 10 && exit 1'] restartPolicy: Never - name: on-failure-container # This container will be restarted on failure. image: docker.io/library/busybox:1.28 command: ['sh', '-c', 'echo "Keep restarting" && sleep 1800 && exit 1'] A Pod with Always restart policy with an init container that only execute once. If the init container fails, the Pod fails. This allows the Pod to fail if the initialization failed, but also keep running once the initialization succeeds:apiVersion: v1 kind: Pod metadata: name: fail-pod-if-init-fails spec: restartPolicy: Always initContainers: - name: init-once # This init container will only try once. If it fails, the pod will fail. image: docker.io/library/busybox:1.28 command: ['sh', '-c', 'echo "Failing initialization" && sleep 10 && exit 1'] restartPolicy: Never containers: - name: main-container # This container will always be restarted once initialization succeeds. image: docker.io/library/busybox:1.28 command: ['sh', '-c', 'sleep 1800 && exit 0'] A Pod with Never restart policy with a container that ignores and restarts on specific exit codes. This is useful to differentiate between restartable errors and non-restartable errors:apiVersion: v1 kind: Pod metadata: name: restart-on-exit-codes spec: restartPolicy: Never containers: - name: restart-on-exit-codes image: docker.io/library/busybox:1.28 command: ['sh', '-c', 'sleep 60 && exit 0'] restartPolicy: Never # Container restart policy must be specified if rules are specified restartPolicyRules: # Only restart the container if it exits with code 42 - action: Restart exitCodes: operator: In values: [42] Restart rules can be used for many more advanced lifecycle management scenarios. Note, restart rules are affected by the same inconsistencies as the regular restart policy. Kubelet restarts, container runtime garbage collection, intermitted connectivity issues with the control plane may cause the state loss and containers may be re-run even when you expect a container not to be restarted.Reduced container restart delayFEATURE STATE: Kubernetes v1.33 [alpha] (enabled by default: false)With the alpha feature gate ReduceDefaultCrashLoopBackOffDecay enabled, container start retries across your cluster will be reduced to begin at 1s (instead of 10s) and increase exponentially by 2x each restart until a maximum delay of 60s (instead of 300s which is 5 minutes).If you use this feature along with the alpha feature KubeletCrashLoopBackOffMax (described below), individual nodes may have different maximum delays.Configurable container restart delayFEATURE STATE: Kubernetes v1.32 [alpha] (enabled by default: false)With the alpha feature gate KubeletCrashLoopBackOffMax enabled, you can reconfigure the maximum delay between container start retries from the default of 300s (5 minutes). This configuration is set per node using kubelet configuration. In your kubelet configuration, under crashLoopBackOff set the maxContainerRestartPeriod field between "1s" and "300s". As described above in Container restart policy, delays on that node will still start at 10s and increase exponentially by 2x each restart, but will now be capped at your configured maximum. If the maxContainerRestartPeriod you configure is less than the default initial value of 10s, the initial delay will instead be set to the configured maximum.See the following kubelet configuration examples:# container restart delays will start at 10s, increasing # 2x each time they are restarted, to a maximum of 100s kind: KubeletConfiguration crashLoopBackOff: maxContainerRestartPeriod: "100s" # delays between container restarts will always be 2s kind: KubeletConfiguration crashLoopBackOff: maxContainerRestartPeriod: "2s" If you use this feature along with the alpha feature ReduceDefaultCrashLoopBackOffDecay (described above), your cluster defaults for initial backoff and maximum backoff will no longer be 10s and 300s, but 1s and 60s. Per node configuration takes precedence over the defaults set by ReduceDefaultCrashLoopBackOffDecay, even if this would result in a node having a longer maximum backoff than other nodes in the cluster.Pod conditionsA Pod has a PodStatus, which has an array of PodConditions through which the Pod has or has not passed. Kubelet manages the following PodConditions:PodScheduled: the Pod has been scheduled to a node.PodReadyToStartContainers: (beta feature; enabled by default) the Pod sandbox has been successfully created and networking configured.ContainersReady: all containers in the Pod are ready.Initialized: all init containers have completed successfully.Ready: the Pod is able to serve requests and should be added to the load balancing pools of all matching Services.DisruptionTarget: the pod is about to be terminated due to a disruption (such as preemption, eviction or garbage-collection).PodResizePending: a pod resize was requested but cannot be applied. See Pod resize status.PodResizeInProgress: the pod is in the process of resizing. See Pod resize status.Field nameDescriptiontypeName of this Pod condition.statusIndicates whether that condition is applicable, with possible values "True", "False", or "Unknown".lastProbeTimeTimestamp of when the Pod condition was last probed.lastTransitionTimeTimestamp for when the Pod last transitioned from one status to another.reasonMachine-readable, UpperCamelCase text indicating the reason for the condition's last transition.messageHuman-readable message indicating details about the last status transition.Pod readinessFEATURE STATE: Kubernetes v1.14 [stable]Your application can inject extra

Pod Lifecycle


This section of the Kubernetes documentation contains pages that show how to do individual tasks. A task page shows how to do a single thing, typically by giving a short sequence of steps.If you would like to write a task page, see Creating a Documentation Pull Request.Install ToolsSet up Kubernetes tools on your computer.Administer a ClusterLearn common tasks for administering a cluster.Configure Pods and ContainersPerform common configuration tasks for Pods and containers.Monitoring, Logging, and DebuggingSet up monitoring and logging to troubleshoot a cluster, or debug a containerized application.Manage Kubernetes ObjectsDeclarative and imperative paradigms for interacting with the Kubernetes API.Managing SecretsManaging confidential settings data using Secrets.Inject Data Into ApplicationsSpecify configuration and other data for the Pods that run your workload.Run ApplicationsRun and manage both stateless and stateful applications.Run JobsRun Jobs using parallel processing.Access Applications in a ClusterConfigure load balancing, port forwarding, or setup firewall or DNS configurations to access applications in a cluster.Extend KubernetesUnderstand advanced ways to adapt your Kubernetes cluster to the needs of your work environment.TLSUnderstand how to protect traffic within your cluster using Transport Layer Security (TLS).Manage Cluster DaemonsPerform common tasks for managing a DaemonSet, such as performing a rolling update.NetworkingLearn how to configure networking for your cluster.Extend kubectl with pluginsExtend kubectl by creating and installing kubectl plugins.Manage HugePagesConfigure and manage huge pages as a schedulable resource in a cluster.Schedule GPUsConfigure and schedule GPUs for use as a resource by nodes in a cluster.

Tasks


kubectl kustomizeSynopsisBuild a set of KRM resources using a 'kustomization.yaml' file. The DIR argument must be a path to a directory containing 'kustomization.yaml', or a git repository URL with a path suffix specifying same with respect to the repository root. If DIR is omitted, '.' is assumed.kubectl kustomize DIR [flags] Examples # Build the current working directory kubectl kustomize # Build some shared configuration directory kubectl kustomize /home/config/production # Build from github kubectl kustomize https://github.com/kubernetes-sigs/kustomize.git/examples/helloWorld?ref=v1.0.6 Options--as-current-useruse the uid and gid of the command executor to run the function in the container--enable-alpha-pluginsenable kustomize plugins--enable-helmEnable use of the Helm chart inflator generator.-e, --env stringsa list of environment variables to be used by functions--helm-api-versions stringsKubernetes api versions used by Helm for Capabilities.APIVersions--helm-command string Default: "helm"helm command (path to executable)--helm-debugEnable debug output from the Helm chart inflator generator.--helm-kube-version stringKubernetes version used by Helm for Capabilities.KubeVersion-h, --helphelp for kustomize--load-restrictor string Default: "LoadRestrictionsRootOnly"if set to 'LoadRestrictionsNone', local kustomizations may load files from outside their root. This does, however, break the relocatability of the kustomization.--mount stringsa list of storage options read from the filesystem--networkenable network access for functions that declare it--network-name string Default: "bridge"the docker network to run the container in-o, --output stringIf specified, write output to this path.Parent Options Inherited--as stringUsername to impersonate for the operation. User could be a regular user or a service account in a namespace.--as-group stringsGroup to impersonate for the operation, this flag can be repeated to specify multiple groups.--as-uid stringUID to impersonate for the operation.--cache-dir string Default: "$HOME/.kube/cache"Default cache directory--certificate-authority stringPath to a cert file for the certificate authority--client-certificate stringPath to a client certificate file for TLS--client-key stringPath to a client key file for TLS--cluster stringThe name of the kubeconfig cluster to use--context stringThe name of the kubeconfig context to use--disable-compressionIf true, opt-out of response compression for all requests to the server--insecure-skip-tls-verifyIf true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure--kubeconfig stringPath to the kubeconfig file to use for CLI requests.--kuberc stringPath to the kuberc file to use for preferences. This can be disabled by exporting KUBECTL_KUBERC=false feature gate or turning off the feature KUBERC=off.--match-server-versionRequire server version to match client version-n, --namespace stringIf present, the namespace scope for this CLI request--password stringPassword for basic authentication to the API server--profile string Default: "none"Name of profile to capture. One of (none|cpu|heap|goroutine|threadcreate|block|mutex)--profile-output string Default: "profile.pprof"Name of the file to write the profile to--request-timeout string Default: "0"The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests.-s, --server stringThe address and port of the Kubernetes API server--storage-driver-buffer-duration duration Default: 1m0sWrites in the storage driver will be buffered for this duration, and committed to the non memory backends as a single transaction--storage-driver-db string Default: "cadvisor"database name--storage-driver-host string Default: "localhost:8086"database host:port--storage-driver-password string Default: "root"database password--storage-driver-secureuse secure connection with database--storage-driver-table string Default: "stats"table name--storage-driver-user string Default: "root"database username--tls-server-name stringServer name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used--token stringBearer token for authentication to the API server--user stringThe name of the kubeconfig user to use--username stringUsername for basic authentication to the API server--version version[=true]--version, --version=raw prints version information and quits; --version=vX.Y.Z... sets the reported version--warnings-as-errorsTreat warnings received from the server as errors and exit with a non-zero exit codeSee Alsokubectl - kubectl controls the Kubernetes cluster managerThis page is automatically generated.If you plan to report an issue with this page, mention that the page is auto-generated in your issue description. The fix may need to happen elsewhere in the Kubernetes project.

kubectl kustomize


ClusterRoleBindingClusterRoleBinding references a ClusterRole, but not contain it.apiVersion: rbac.authorization.k8s.io/v1import "k8s.io/api/rbac/v1"ClusterRoleBindingClusterRoleBinding references a ClusterRole, but not contain it. It can reference a ClusterRole in the global namespace, and adds who information via Subject.apiVersion: rbac.authorization.k8s.io/v1kind: ClusterRoleBindingmetadata (ObjectMeta)Standard object's metadata.roleRef (RoleRef), requiredRoleRef can only reference a ClusterRole in the global namespace. If the RoleRef cannot be resolved, the Authorizer must return an error. This field is immutable.RoleRef contains information that points to the role being usedroleRef.apiGroup (string), requiredAPIGroup is the group for the resource being referencedroleRef.kind (string), requiredKind is the type of resource being referencedroleRef.name (string), requiredName is the name of resource being referencedsubjects ([]Subject)Atomic: will be replaced during a mergeSubjects holds references to the objects the role applies to.Subject contains a reference to the object or user identities a role binding applies to. This can either hold a direct API object reference, or a value for non-objects such as user and group names.subjects.kind (string), requiredKind of object being referenced. Values defined by this API group are "User", "Group", and "ServiceAccount". If the Authorizer does not recognized the kind value, the Authorizer should report an error.subjects.name (string), requiredName of the object being referenced.subjects.apiGroup (string)APIGroup holds the API group of the referenced subject. Defaults to "" for ServiceAccount subjects. Defaults to "rbac.authorization.k8s.io" for User and Group subjects.subjects.namespace (string)Namespace of the referenced object. If the object kind is non-namespace, such as "User" or "Group", and this value is not empty the Authorizer should report an error.ClusterRoleBindingListClusterRoleBindingList is a collection of ClusterRoleBindingsapiVersion: rbac.authorization.k8s.io/v1kind: ClusterRoleBindingListmetadata (ListMeta)Standard object's metadata.items ([]ClusterRoleBinding), requiredItems is a list of ClusterRoleBindingsOperationsget read the specified ClusterRoleBindingHTTP RequestGET /apis/rbac.authorization.k8s.io/v1/clusterrolebindings/{name}Parametersname (in path): string, requiredname of the ClusterRoleBindingpretty (in query): stringprettyResponse200 (ClusterRoleBinding): OK401: Unauthorizedlist list or watch objects of kind ClusterRoleBindingHTTP RequestGET /apis/rbac.authorization.k8s.io/v1/clusterrolebindingsParametersallowWatchBookmarks (in query): booleanallowWatchBookmarkscontinue (in query): stringcontinuefieldSelector (in query): stringfieldSelectorlabelSelector (in query): stringlabelSelectorlimit (in query): integerlimitpretty (in query): stringprettyresourceVersion (in query): stringresourceVersionresourceVersionMatch (in query): stringresourceVersionMatchsendInitialEvents (in query): booleansendInitialEventstimeoutSeconds (in query): integertimeoutSecondswatch (in query): booleanwatchResponse200 (ClusterRoleBindingList): OK401: Unauthorizedcreate create a ClusterRoleBindingHTTP RequestPOST /apis/rbac.authorization.k8s.io/v1/clusterrolebindingsParametersbody: ClusterRoleBinding, requireddryRun (in query): stringdryRunfieldManager (in query): stringfieldManagerfieldValidation (in query): stringfieldValidationpretty (in query): stringprettyResponse200 (ClusterRoleBinding): OK201 (ClusterRoleBinding): Created202 (ClusterRoleBinding): Accepted401: Unauthorizedupdate replace the specified ClusterRoleBindingHTTP RequestPUT /apis/rbac.authorization.k8s.io/v1/clusterrolebindings/{name}Parametersname (in path): string, requiredname of the ClusterRoleBindingbody: ClusterRoleBinding, requireddryRun (in query): stringdryRunfieldManager (in query): stringfieldManagerfieldValidation (in query): stringfieldValidationpretty (in query): stringprettyResponse200 (ClusterRoleBinding): OK201 (ClusterRoleBinding): Created401: Unauthorizedpatch partially update the specified ClusterRoleBindingHTTP RequestPATCH /apis/rbac.authorization.k8s.io/v1/clusterrolebindings/{name}Parametersname (in path): string, requiredname of the ClusterRoleBindingbody: Patch, requireddryRun (in query): stringdryRunfieldManager (in query): stringfieldManagerfieldValidation (in query): stringfieldValidationforce (in query): booleanforcepretty (in query): stringprettyResponse200 (ClusterRoleBinding): OK201 (ClusterRoleBinding): Created401: Unauthorizeddelete delete a ClusterRoleBindingHTTP RequestDELETE /apis/rbac.authorization.k8s.io/v1/clusterrolebindings/{name}Parametersname (in path): string, requiredname of the ClusterRoleBindingbody: DeleteOptionsdryRun (in query): stringdryRungracePeriodSeconds (in query): integergracePeriodSecondsignoreStoreReadErrorWithClusterBreakingPotential (in query): booleanignoreStoreReadErrorWithClusterBreakingPotentialpretty (in query): stringprettypropagationPolicy (in query): stringpropagationPolicyResponse200 (Status): OK202 (Status): Accepted401: Unauthorizeddeletecollection delete collection of ClusterRoleBindingHTTP RequestDELETE /apis/rbac.authorization.k8s.io/v1/clusterrolebindingsParametersbody: DeleteOptionscontinue (in query): stringcontinuedryRun (in query): stringdryRunfieldSelector (in query): stringfieldSelectorgracePeriodSeconds (in query): integergracePeriodSecondsignoreStoreReadErrorWithClusterBreakingPotential (in query): booleanignoreStoreReadErrorWithClusterBreakingPotentiallabelSelector (in query): stringlabelSelectorlimit (in query): integerlimitpretty (in query): stringprettypropagationPolicy (in query): stringpropagationPolicyresourceVersion (in query): stringresourceVersionresourceVersionMatch (in query): stringresourceVersionMatchsendInitialEvents (in query): booleansendInitialEventstimeoutSeconds (in query): integertimeoutSecondsResponse200 (Status): OK401: UnauthorizedThis page is automatically generated.If you plan to report an issue with this page, mention that the page is auto-generated in your issue description. The fix may need to happen elsewhere in the Kubernetes project.

ClusterRoleBinding


This section lists the different ways to set up and run Kubernetes. When you install Kubernetes, choose an installation type based on: ease of maintenance, security, control, available resources, and expertise required to operate and manage a cluster.You can download Kubernetes to deploy a Kubernetes cluster on a local machine, into the cloud, or for your own datacenter.Several Kubernetes components such as kube-apiserver or kube-proxy can also be deployed as container images within the cluster.It is recommended to run Kubernetes components as container images wherever that is possible, and to have Kubernetes manage those components. Components that run containers - notably, the kubelet - can't be included in this category.If you don't want to manage a Kubernetes cluster yourself, you could pick a managed service, including certified platforms. There are also other standardized and custom solutions across a wide range of cloud and bare metal environments.Learning environmentIf you're learning Kubernetes, use the tools supported by the Kubernetes community, or tools in the ecosystem to set up a Kubernetes cluster on a local machine. See Install tools.Production environmentWhen evaluating a solution for a production environment, consider which aspects of operating a Kubernetes cluster (or abstractions) you want to manage yourself and which you prefer to hand off to a provider.For a cluster you're managing yourself, the officially supported tool for deploying Kubernetes is kubeadm.What's nextDownload KubernetesDownload and install tools including kubectlSelect a container runtime for your new clusterLearn about best practices for cluster setupKubernetes is designed for its control plane to run on Linux. Within your cluster you can run applications on Linux or other operating systems, including Windows.Learn to set up clusters with Windows nodes

Getting started