# Versions in CustomResourceDefinitions

Versions in CustomResourceDefinitionsThis page explains how to add versioning information to CustomResourceDefinitions, to indicate the stability level of your CustomResourceDefinitions or advance your API to a new version with conversion between API representations. It also describes how to upgrade an object from one version to another.Before you beginYou need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. It is recommended to run this tutorial on a cluster with at least two nodes that are not acting as control plane hosts. If you do not already have a cluster, you can create one by using minikube or you can use one of these Kubernetes playgrounds:iximiuz LabsKillercodaKodeKloudPlay with KubernetesYou should have an initial understanding of custom resources.Your Kubernetes server must be at or later than version v1.16.To check the version, enter kubectl version.OverviewThe CustomResourceDefinition API provides a workflow for introducing and upgrading to new versions of a CustomResourceDefinition.When a CustomResourceDefinition is created, the first version is set in the CustomResourceDefinition spec.versions list to an appropriate stability level and a version number. For example v1beta1 would indicate that the first version is not yet stable. All custom resource objects will initially be stored at this version.Once the CustomResourceDefinition is created, clients may begin using the v1beta1 API.Later it might be necessary to add new version such as v1.Adding a new version:Pick a conversion strategy. Since custom resource objects need the ability to be served at both versions, that means they will sometimes be served in a different version than the one stored. To make this possible, the custom resource objects must sometimes be converted between the version they are stored at and the version they are served at. If the conversion involves schema changes and requires custom logic, a conversion webhook should be used. If there are no schema changes, the default None conversion strategy may be used and only the apiVersion field will be modified when serving different versions.If using conversion webhooks, create and deploy the conversion webhook. See the Webhook conversion for more details.Update the CustomResourceDefinition to include the new version in the spec.versions list with served:true. Also, set spec.conversion field to the selected conversion strategy. If using a conversion webhook, configure spec.conversion.webhookClientConfig field to call the webhook.Once the new version is added, clients may incrementally migrate to the new version. It is perfectly safe for some clients to use the old version while others use the new version.Migrate stored objects to the new version:See the upgrade existing objects to a new stored version section.It is safe for clients to use both the old and new version before, during and after upgrading the objects to a new stored version.Removing an old version:Ensure all clients are fully migrated to the new version. The kube-apiserver logs can be reviewed to help identify any clients that are still accessing via the old version.Set served to false for the old version in the spec.versions list. If any clients are still unexpectedly using the old version they may begin reporting errors attempting to access the custom resource objects at the old version. If this occurs, switch back to using served:true on the old version, migrate the remaining clients to the new version and repeat this step.Ensure the upgrade of existing objects to the new stored version step has been completed.Verify that the storage is set to true for the new version in the spec.versions list in the CustomResourceDefinition.Verify that the old version is no longer listed in the CustomResourceDefinition status.storedVersions.Remove the old version from the CustomResourceDefinition spec.versions list.Drop conversion support for the old version in conversion webhooks.Specify multiple versionsThe CustomResourceDefinition API versions field can be used to support multiple versions of custom resources that you have developed. Versions can have different schemas, and conversion webhooks can convert custom resources between versions. Webhook conversions should follow the Kubernetes API conventions wherever applicable. Specifically, See the API change documentation for a set of useful gotchas and suggestions.Note:In apiextensions.k8s.io/v1beta1, there was a version field instead of versions. The version field is deprecated and optional, but if it is not empty, it must match the first item in the versions field.This example shows a CustomResourceDefinition with two versions. For the first example, the assumption is all versions share the same schema with no conversion between them. The comments in the YAML provide more context.apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: # name must match the spec fields below, and be in the form: <plural>.<group> name: crontabs.example.com spec: # group name to use for REST API: /apis/<group>/<version> group: example.com # list of versions supported by this CustomResourceDefinition versions: - name: v1beta1 # Each version can be enabled/disabled by Served flag. served: true # One and only one version must be marked as the storage version. storage: true # A schema is required schema: openAPIV3Schema: type: object properties: host: type: string port: type: string - name: v1 served: true storage: false schema: openAPIV3Schema: type: object properties: host: type: string port: type: string # The conversion section is introduced in Kubernetes 1.13+ with a default value of # None conversion (strategy sub-field set to None). conversion: # None conversion assumes the same schema for all versions and only sets the apiVersion # field of custom resources to the proper value strategy: None # either Namespaced or Cluster scope: Namespaced names: # plural name to be used in the URL: /apis/<group>/<version>/<plural> plural: crontabs # singular name to be used as an alias on the CLI and for display singular: crontab # kind is normally the CamelCased singular type. Your resource manifests use this. kind: CronTab # shortNames allow shorter string to match your resource on the CLI shortNames: - ct # Deprecated in v1.16 in favor of apiextensions.k8s.io/v1 apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata: # name must match the spec fields below, and be in the form: <plural>.<group> name: crontabs.example.com spec: # group name to use for REST API: /apis/<group>/<version> group: example.com # list of versions supported by this CustomResourceDefinition versions: - name: v1beta1 # Each version can be enabled/disabled by Served flag. served: true # One and only one version must be marked as the storage version. storage: true - name: v1 served: true storage: false validation: openAPIV3Schema: type: object properties: host: type: string port: type: string # The conversion section is introduced in Kubernetes 1.13+ with a default value of # None conversion (strategy sub-field set to None). conversion: # None conversion assumes the same schema for all versions and only sets the apiVersion # field of custom resources to the proper value strategy: None # either Namespaced or Cluster scope: Namespaced names: # plural name to be used in the URL: /apis/<group>/<version>/<plural> plural: crontabs # singular name to be used as an alias on the CLI and for display singular: crontab # kind is normally the PascalCased singular type. Your resource manifests use this. kind: CronTab # shortNames allow shorter string to match your resource on the CLI shortNames: - ct You can save the CustomResourceDefinition in a YAML file, then use kubectl apply to create it.kubectl apply -f my-versioned-crontab.yaml After creation, the API server starts to serve each enabled version at an HTTP REST endpoint. In the above example, the API versions are available at /apis/example.com/v1beta1 and /apis/example.com/v1.Version priorityRegardless of the order in which versions are defined in a CustomResourceDefinition, the version with the highest priority is used by kubectl as the default version to access objects. The priority is determined by parsing the name field to determine the version number, the stability (GA, Beta, or Alpha), and the sequence within that stability level.The algorithm used for sorting the versions is designed to sort versions in the same way that the Kubernetes project sorts Kubernetes versions. Versions start with a v followed by a number, an optional beta or alpha designation, and optional additional numeric versioning information. Broadly, a version string might look like v2 or v2beta1. Versions are sorted using the following algorithm:Entries that follow Kubernetes version patterns are sorted before those that do not.For entries that follow Kubernetes version patterns, the numeric portions of the version string is sorted largest to smallest.If the strings beta or alpha follow the first numeric portion, they sorted in that order, after the equivalent string without the beta or alpha suffix (which is presumed to be the GA version).If another number follows the beta, or alpha, those numbers are also sorted from largest to smallest.Strings that don't fit the above format are sorted alphabetically and the numeric portions are not treated specially. Notice that in the example below, foo1 is sorted above foo10. This is different from the sorting of the numeric portion of entries that do follow the Kubernetes version patterns.This might make sense if you look at the following sorted version list:- v10 - v2 - v1 - v11beta2 - v10beta3 - v3beta1 - v12alpha1 - v11alpha2 - foo1 - foo10 For the example in Specify multiple versions, the version sort order is v1, followed by v1beta1. This causes the kubectl command to use v1 as the default version unless the provided object specifies the version.Version deprecationFEATURE STATE: Kubernetes v1.19 [stable]Starting in v1.19, a CustomResourceDefinition can indicate a particular version of the resource it defines is deprecated. When API requests to a deprecated version of that resource are made, a warning message is returned in the API response as a header. The warning message for each deprecated version of the resource can be customized if desired.A customized warning message should indicate the deprecated API group, version, and kind, and should indicate what API group, version, and kind should be used instead, if applicable.apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition name: crontabs.example.com spec: group: example.com names: plural: crontabs singular: crontab kind: CronTab scope: Namespaced versions: - name: v1alpha1 served: true storage: false # This indicates the v1alpha1 version of the custom resource is deprecated. # API requests to this version receive a warning header in the server response. deprecated: true # This overrides the default warning returned to API clients making v1alpha1 API requests. deprecationWarning: "example.com/v1alpha1 CronTab is deprecated; see http://example.com/v1alpha1-v1 for instructions to migrate to example.com/v1 CronTab" schema: ... - name: v1beta1 served: true # This indicates the v1beta1 version of the custom resource is deprecated. # API requests to this version receive a warning header in the server response. # A default warning message is returned for this version. deprecated: true schema: ... - name: v1 served: true storage: true schema: ... # Deprecated in v1.16 in favor of apiextensions.k8s.io/v1 apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata: name: crontabs.example.com spec: group: example.com names: plural: crontabs singular: crontab kind: CronTab scope: Namespaced validation: ... versions: - name: v1alpha1 served: true storage: false # This indicates the v1alpha1 version of the custom resource is deprecated. # API requests to this version receive a warning header in the server response. deprecated: true # This overrides the default warning returned to API clients making v1alpha1 API requests. deprecationWarning: "example.com/v1alpha1 CronTab is deprecated; see http://example.com/v1alpha1-v1 for instructions to migrate to example.com/v1 CronTab" - name: v1beta1 served: true # This indicates the v1beta1 version of the custom resource is deprecated. # API requests to this version receive a warning header in the server response. # A default warning message is returned for this version. deprecated: true - name: v1 served: true storage: true Version removalAn older API version cannot be dropped from a CustomResourceDefinition manifest until existing stored data has been migrated to the newer API version for all clusters that served the older version of the custom resource, and the old version is removed from the status.storedVersions of the CustomResourceDefinition.apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition name: crontabs.example.com spec: group: example.com names: plural: crontabs singular: crontab kind: CronTab scope: Namespaced versions: - name: v1beta1 # This indicates the v1beta1 version of the custom resource is no longer served. # API requests to this version receive a not found error in the server response. served: false schema: ... - name: v1 served: true # The new served version should be set as the storage version storage: true schema: ... Webhook conversionFEATURE STATE: Kubernetes v1.16 [stable]Note:Webhook conversion is available as beta since 1.15, and as alpha since Kubernetes 1.13. The CustomResourceWebhookConversion feature must be enabled, which is the case automatically for many clusters for beta features. Please refer to the feature gate documentation for more information.The above example has a None conversion between versions which only sets the apiVersion field on conversion and does not change the rest of the object. The API server also supports webhook conversions that call an external service in case a conversion is required. For example when:custom resource is requested in a different version than stored version.Watch is created in one version but the changed object is stored in another version.custom resource PUT request is in a different version than storage version.To cover all of these cases and to optimize conversion by the API server, the conversion requests may contain multiple objects in order to minimize the external calls. The webhook should perform these conversions independently.Write a conversion webhook serverPlease refer to the implementation of the custom resource conversion webhook server that is validated in a Kubernetes e2e test. The webhook handles the ConversionReview requests sent by the API servers, and sends back conversion results wrapped in ConversionResponse. Note that the request contains a list of custom resources that need to be converted independently without changing the order of objects. The example server is organized in a way to be reused for other conversions. Most of the common code are located in the framework file that leaves only one function to be implemented for different conversions.Note:The example conversion webhook server leaves the ClientAuth field empty, which defaults to NoClientCert. This means that the webhook server does not authenticate the identity of the clients, supposedly API servers. If you need mutual TLS or other ways to authenticate the clients, see how to authenticate API servers.Permissible mutationsA conversion webhook must not mutate anything inside of metadata of the converted object other than labels and annotations. Attempted changes to name, UID and namespace are rejected and fail the request which caused the conversion. All other changes are ignored.Deploy the conversion webhook serviceDocumentation for deploying the conversion webhook is the same as for the admission webhook example service. The assumption for


# Concepts

The Concepts section helps you learn about the parts of the Kubernetes system and the abstractions Kubernetes uses to represent your cluster, and helps you obtain a deeper understanding of how Kubernetes works.OverviewKubernetes is a portable, extensible, open source platform for managing containerized workloads and services, that facilitates both declarative configuration and automation. It has a large, rapidly growing ecosystem. Kubernetes services, support, and tools are widely available.Cluster ArchitectureThe architectural concepts behind Kubernetes.ContainersTechnology for packaging an application along with its runtime dependencies.WorkloadsUnderstand Pods, the smallest deployable compute object in Kubernetes, and the higher-level abstractions that help you to run them.Services, Load Balancing, and NetworkingConcepts and resources behind networking in Kubernetes.StorageWays to provide both long-term and temporary storage to Pods in your cluster.ConfigurationResources that Kubernetes provides for configuring Pods.SecurityConcepts for keeping your cloud-native workload secure.PoliciesManage security and best-practices with policies.Scheduling, Preemption and EvictionCluster AdministrationLower-level detail relevant to creating or administering a Kubernetes cluster.Windows in KubernetesKubernetes supports nodes that run Microsoft Windows.Extending KubernetesDifferent ways to change the behavior of your Kubernetes cluster.


# Policies

This is the multi-page printable view of this section. Click here to print.Return to the regular view of this page.PoliciesManage security and best-practices with policies.1: Limit Ranges2: Resource Quotas3: Process ID Limits And Reservations4: Node Resource ManagersKubernetes policies are configurations that manage other configurations or runtime behaviors. Kubernetes offers various forms of policies, described below:Apply policies using API objectsSome API objects act as policies. Here are some examples:NetworkPolicies can be used to restrict ingress and egress traffic for a workload.LimitRanges manage resource allocation constraints across different object kinds.ResourceQuotas limit resource consumption for a namespace.Apply policies using admission controllersAn admission controller runs in the API server and can validate or mutate API requests. Some admission controllers act to apply policies. For example, the AlwaysPullImages admission controller modifies a new Pod to set the image pull policy to Always.Kubernetes has several built-in admission controllers that are configurable via the API server --enable-admission-plugins flag.Details on admission controllers, with the complete list of available admission controllers, are documented in a dedicated section:Admission ControllersApply policies using ValidatingAdmissionPolicyValidating admission policies allow configurable validation checks to be executed in the API server using the Common Expression Language (CEL). For example, a ValidatingAdmissionPolicy can be used to disallow use of the latest image tag.A ValidatingAdmissionPolicy operates on an API request and can be used to block, audit, and warn users about non-compliant configurations.Details on the ValidatingAdmissionPolicy API, with examples, are documented in a dedicated section:Validating Admission PolicyApply policies using dynamic admission controlDynamic admission controllers (or admission webhooks) run outside the API server as separate applications that register to receive webhooks requests to perform validation or mutation of API requests.Dynamic admission controllers can be used to apply policies on API requests and trigger other policy-based workflows. A dynamic admission controller can perform complex checks including those that require retrieval of other cluster resources and external data. For example, an image verification check can lookup data from OCI registries to validate the container image signatures and attestations.Details on dynamic admission control are documented in a dedicated section:Dynamic Admission ControlImplementationsNote: This section links to third party projects that provide functionality required by Kubernetes. The Kubernetes project authors aren't responsible for these projects, which are listed alphabetically. To add a project to this list, read the content guide before submitting a change. More information.Dynamic Admission Controllers that act as flexible policy engines are being developed in the Kubernetes ecosystem, such as:KubewardenKyvernoOPA GatekeeperPolarisApply policies using Kubelet configurationsKubernetes allows configuring the Kubelet on each worker node. Some Kubelet configurations act as policies:Process ID limits and reservations are used to limit and reserve allocatable PIDs.Node Resource Managers can manage compute, memory, and device resources for latency-critical and high-throughput workloads.1 - Limit RangesBy default, containers run with unbounded compute resources on a Kubernetes cluster. Using Kubernetes resource quotas, administrators (also termed cluster operators) can restrict consumption and creation of cluster resources (such as CPU time, memory, and persistent storage) within a specified namespace. Within a namespace, a Pod can consume as much CPU and memory as is allowed by the ResourceQuotas that apply to that namespace. As a cluster operator, or as a namespace-level administrator, you might also be concerned about making sure that a single object cannot monopolize all available resources within a namespace.A LimitRange is a policy to constrain the resource allocations (limits and requests) that you can specify for each applicable object kind (such as Pod or PersistentVolumeClaim) in a namespace.A LimitRange provides constraints that can:Enforce minimum and maximum compute resources usage per Pod or Container in a namespace.Enforce minimum and maximum storage request per PersistentVolumeClaim in a namespace.Enforce a ratio between request and limit for a resource in a namespace.Set default request/limit for compute resources in a namespace and automatically inject them to Containers at runtime.Kubernetes constrains resource allocations to Pods in a particular namespace whenever there is at least one LimitRange object in that namespace.The name of a LimitRange object must be a valid DNS subdomain name.Constraints on resource limits and requestsThe administrator creates a LimitRange in a namespace.Users create (or try to create) objects in that namespace, such as Pods or PersistentVolumeClaims.First, the LimitRange admission controller applies default request and limit values for all Pods (and their containers) that do not set compute resource requirements.Second, the LimitRange tracks usage to ensure it does not exceed resource minimum, maximum and ratio defined in any LimitRange present in the namespace.If you attempt to create or update an object (Pod or PersistentVolumeClaim) that violates a LimitRange constraint, your request to the API server will fail with anHTTP status code 403 Forbidden and a message explaining the constraint that has been violated.If you add a LimitRange in a namespace that applies to compute-related resources such as cpu and memory, you must specify requests or limits for those values. Otherwise, the system may reject Pod creation.LimitRange validations occur only at Pod admission stage, not on running Pods. If you add or modify a LimitRange, the Pods that already exist in that namespace continue unchanged.If two or more LimitRange objects exist in the namespace, it is not deterministic which default value will be applied.LimitRange and admission checks for PodsA LimitRange does not check the consistency of the default values it applies. This means that a default value for the limit that is set by LimitRange may be less than the request value specified for the container in the spec that a client submits to the API server. If that happens, the final Pod will not be schedulable.For example, you define a LimitRange with below manifest:Note:The following examples operate within the default namespace of your cluster, as the namespace parameter is undefined and the LimitRange scope is limited to the namespace level. This implies that any references or operations within these examples will interact with elements within the default namespace of your cluster. You can override the operating namespace by configuring namespace in the metadata.namespace field.concepts/policy/limit-range/problematic-limit-range.yaml apiVersion: v1 kind: LimitRange metadata: name: cpu-resource-constraint spec: limits: - default: # this section defines default limits cpu: 500m defaultRequest: # this section defines default requests cpu: 500m max: # max and min define the limit range cpu: "1" min: cpu: 100m type: Container along with a Pod that declares a CPU resource request of 700m, but not a limit:concepts/policy/limit-range/example-conflict-with-limitrange-cpu.yaml apiVersion: v1 kind: Pod metadata: name: example-conflict-with-limitrange-cpu spec: containers: - name: demo image: registry.k8s.io/pause:3.8 resources: requests: cpu: 700m then that Pod will not be scheduled, failing with an error similar to:Pod "example-conflict-with-limitrange-cpu" is invalid: spec.containers[0].resources.requests: Invalid value: "700m": must be less than or equal to cpu limit If you set both request and limit, then that new Pod will be scheduled successfully even with the same LimitRange in place:concepts/policy/limit-range/example-no-conflict-with-limitrange-cpu.yaml apiVersion: v1 kind: Pod metadata: name: example-no-conflict-with-limitrange-cpu spec: containers: - name: demo image: registry.k8s.io/pause:3.8 resources: requests: cpu: 700m limits: cpu: 700m Example resource constraintsExamples of policies that could be created using LimitRange are:In a 2 node cluster with a capacity of 8 GiB RAM and 16 cores, constrain Pods in a namespace to request 100m of CPU with a max limit of 500m for CPU and request 200Mi for Memory with a max limit of 600Mi for Memory.Define default CPU limit and request to 150m and memory default request to 300Mi for Containers started with no cpu and memory requests in their specs.In the case where the total limits of the namespace is less than the sum of the limits of the Pods/Containers, there may be contention for resources. In this case, the Containers or Pods will not be created.Neither contention nor changes to a LimitRange will affect already created resources.What's nextFor examples on using limits, see:how to configure minimum and maximum CPU constraints per namespace.how to configure minimum and maximum Memory constraints per namespace.how to configure default CPU Requests and Limits per namespace.how to configure default Memory Requests and Limits per namespace.how to configure minimum and maximum Storage consumption per namespace.a detailed example on configuring quota per namespace.Refer to the LimitRanger design document for context and historical information.2 - Resource QuotasWhen several users or teams share a cluster with a fixed number of nodes, there is a concern that one team could use more than its fair share of resources.Resource quotas are a tool for administrators to address this concern.A resource quota, defined by a ResourceQuota object, provides constraints that limit aggregate resource consumption per namespace. A ResourceQuota can also limit the quantity of objects that can be created in a namespace by API kind, as well as the total amount of infrastructure resources that may be consumed by API objects found in that namespace.Caution:Neither contention nor changes to quota will affect already created resources.How Kubernetes ResourceQuotas workResourceQuotas work like this:Different teams work in different namespaces. This separation can be enforced with RBAC or any other authorization mechanism.A cluster administrator creates at least one ResourceQuota for each namespace.To make sure the enforcement stays enforced, the cluster administrator should also restrict access to delete or update that ResourceQuota; for example, by defining a ValidatingAdmissionPolicy.Users create resources (pods, services, etc.) in the namespace, and the quota system tracks usage to ensure it does not exceed hard resource limits defined in a ResourceQuota.You can apply a scope to a ResourceQuota to limit where it applies,If creating or updating a resource violates a quota constraint, the control plane rejects that request with HTTP status code 403 Forbidden. The error includes a message explaining the constraint that would have been violated.If quotas are enabled in a namespace for resource such as cpu and memory, users must specify requests or limits for those values when they define a Pod; otherwise, the quota system may reject pod creation.The resource quota walkthrough shows an example of how to avoid this problem.Note:You can define a LimitRange to force defaults on pods that make no compute resource requirements (so that users don't have to remember to do that).You often do not create Pods directly; for example, you more usually create a workload management object such as a Deployment. If you create a Deployment that tries to use more resources than are available, the creation of the Deployment (or other workload management object) succeeds, but the Deployment may not be able to get all of the Pods it manages to exist. In that case you can check the status of the Deployment, for example with kubectl describe, to see what has happened.For cpu and memory resources, ResourceQuotas enforce that every (new) pod in that namespace sets a limit for that resource. If you enforce a resource quota in a namespace for either cpu or memory, you and other clients, must specify either requests or limits for that resource, for every new Pod you submit. If you don't, the control plane may reject admission for that Pod.For other resources: ResourceQuota works and will ignore pods in the namespace without setting a limit or request for that resource. It means that you can create a new pod without limit/request for ephemeral storage if the resource quota limits the ephemeral storage of this namespace.You can use a LimitRange to automatically set a default request for these resources.The name of a ResourceQuota object must be a valid DNS subdomain name.Examples of policies that could be created using namespaces and quotas are:In a cluster with a capacity of 32 GiB RAM, and 16 cores, let team A use 20 GiB and 10 cores, let B use 10GiB and 4 cores, and hold 2GiB and 2 cores in reserve for future allocation.Limit the "testing" namespace to using 1 core and 1GiB RAM. Let the "production" namespace use any amount.In the case where the total capacity of the cluster is less than the sum of the quotas of the namespaces, there may be contention for resources. This is handled on a first-come-first-served basis.Enabling Resource QuotaResourceQuota support is enabled by default for many Kubernetes distributions. It is enabled when the API server --enable-admission-plugins= flag has ResourceQuota as one of its arguments.A resource quota is enforced in a particular namespace when there is a ResourceQuota in that namespace.Types of resource quotaThe ResourceQuota mechanism lets you enforce different kinds of limits. This section describes the types of limit that you can enforce.Quota for infrastructure resourcesYou can limit the total sum of compute resources that can be requested in a given namespace.The following resource types are supported:Resource NameDescriptionlimits.cpuAcross all pods in a non-terminal state, the sum of CPU limits cannot exceed this value.limits.memoryAcross all pods in a non-terminal state, the sum of memory limits cannot exceed this value.requests.cpuAcross all pods in a non-terminal state, the sum of CPU requests cannot exceed this value.requests.memoryAcross all pods in a non-terminal state, the sum of memory requests cannot exceed this value.hugepages-<size>Across all pods in a non-terminal state, the number of huge page requests of the specified size cannot exceed this value.cpuSame as requests.cpumemorySame as requests.memoryQuota for extended resourcesIn addition to the resources mentioned above, in release 1.10, quota support for extended resources is added.As overcommit is not allowed for extended resources, it makes no sense to specify both requests and limits for the same extended resource in a quota. So for extended resources, only quota items with prefix requests. are allowed.Take the GPU resource as an example, if the resource name is nvidia.com/gpu, and you want to limit the total number of GPUs requested in a namespace to 4, you can define a quota as follows:requests.nvidia.com/gpu: 4See Viewing and Setting Quotas for more details.Quota for storageYou can limit the total sum of storage for volumes that can be requested in a given namespace.In addition, you can limit consumption of storage resources based on associated StorageClass.Resource NameDescriptionrequests.storageAcross all persistent volume claims, the sum of storage requests cannot exceed this value.persistentvolumeclaimsThe total number of PersistentVolumeClaims that can exist in the namespace.<storage-class-name>.storageclass.storage.k8s.io/requests.storageAcross all persistent volume claims associated with the <storage-class-name>, the sum of storage requests cannot exceed this value.<storage-class-name>.storageclass.storage.k8s.io/persistentvolumeclaimsAcross all persistent volume claims associated with the <storage-class-name>, the total number of persistent volume claims that can exist in the namespace.For example, if you want to quota storage with gold StorageClass separate from a bronze StorageClass, you can define a quota as follows:gold.storageclass.storage.k8s.io/requests.storage: 500Gibronze.storageclass.storage.k8s.io/requests.storage: 100GiQuota for local ephemeral storageFEATURE STATE: Kubernetes v1.8 [alpha]Resource NameDescriptionrequests.ephemeral-storageAcross all pods in the namespace, the sum of local ephemeral storage requests cannot exceed this value.limits.ephemeral-storageAcross all pods in the namespace, the sum of local ephemeral storage limits cannot exceed this value.ephemeral-storageSame as requests.ephemeral-storage.Note:When using a CRI container runtime, container logs will count against the ephemeral storage quota. This can result in the unexpected eviction of pods that have exhausted their storage quotas.Refer to Logging Architecture for details.Quota on object countYou can set quota for the total number of one particular resource kind in the Kubernetes API, using the following syntax:count/<resource>.<group> for resources from non-core API groupscount/<resource> for resources from the core API groupFor example, the PodTemplate API is in the core API group and so if you want to limit the number of PodTemplate objects in a namespace, you use count/podtemplates.These types of quotas are useful to protect against exhaustion of control plane storage. For example, you may want to limit the number of Secrets in a server given their large size. Too many Secrets in a cluster can actually prevent servers and controllers from starting. You can set a quota for Jobs to protect against a poorly configured CronJob. CronJobs that create too many Jobs in a namespace can lead to a denial of service.If you define a quota this way, it applies to Kubernetes' APIs that are part of the API server, and to any custom resources backed by a CustomResourceDefinition. For example, to create a quota on a widgets custom resource in the example.com API group, use count/widgets.example.com. If you use API aggregation to add additional, custom APIs that are not defined as CustomResourceDefinitions, the core Kubernetes control plane does not enforce quota for the aggregated API. The extension API server is expected to provide quota enforcement if that's appropriate for the custom API.Generic syntaxThis is a list of common examples of object kinds that you may want to put under object count quota, listed by the configuration string that you would use.count/podscount/persistentvolumeclaimscount/servicescount/secretscount/configmapscount/deployments.appscount/replicasets.appscount/statefulsets.appscount/jobs.batchcount/cronjobs.batchSpecialized syntaxThere is another syntax only to set the same type of quota, that only works for certain API kinds. The following types are supported:Resource NameDescriptionconfigmapsThe total number of ConfigMaps that can exist in the namespace.persistentvolumeclaimsThe total number of PersistentVolumeClaims that can exist in the namespace.podsThe total number of Pods in a non-terminal state that can exist in the namespace. A pod is in a terminal state if .status.phase in (Failed, Succeeded) is true.replicationcontrollersThe total number of ReplicationControllers that can exist in the namespace.resourcequotasThe total number of ResourceQuotas that can exist in the namespace.servicesThe total number of Services that can exist in the namespace.services.loadbalancersThe total number of Services of type LoadBalancer that can exist in the namespace.services.nodeportsThe total number of NodePorts allocated to Services of type NodePort or LoadBalancer that can exist in the namespace.secretsThe total number of Secrets that can exist in the namespace.For example, pods quota counts and enforces a maximum on the number of pods created in a single namespace that are not terminal. You might want to set a pods quota on a namespace to avoid the case where a user creates many small pods and exhausts the cluster's supply of Pod IPs.You can find more examples on Viewing and Setting Quotas.Quota ScopesEach quota can have an associated set of scopes. A quota will only measure usage for a resource if it matches the intersection of enumerated scopes.When a scope is added to the quota, it limits the number of resources it supports to those that pertain to the scope. Resources specified on the quota outside of the allowed set results in a validation error.ScopeDescriptionTerminatingMatch pods where .spec.activeDeadlineSeconds >= 0NotTerminatingMatch pods where .spec.activeDeadlineSeconds is nilBestEffortMatch pods that have best effort quality of service.NotBestEffortMatch pods that do not have best effort quality of service.PriorityClassMatch pods that references the specified priority class.CrossNamespacePodAffinityMatch pods that have cross-namespace pod (anti)affinity terms.VolumeAttributesClassMatch persistentvolumeclaims that references the specified volume attributes class.The BestEffort scope restricts a quota to tracking the following resource:podsThe Terminating, NotTerminating, NotBestEffort and PriorityClass scopes restrict a quota to tracking the following resources:podscpumemoryrequests.cpurequests.memorylimits.cpulimits.memoryNote that you cannot specify both the Terminating and the NotTerminating scopes in the same quota, and you cannot specify both the BestEffort and NotBestEffort scopes in the same quota either.The scopeSelector supports the following values in the operator field:InNotInExistsDoesNotExistWhen using one of the following values as the scopeName when defining the scopeSelector, the operator must be Exists.TerminatingNotTerminatingBestEffortNotBestEffortIf the operator is In or NotIn, the values field must have at least one value. For example: scopeSelector: matchExpressions: - scopeName: PriorityClass operator: In values: - middle If the operator is Exists or DoesNotExist, the values field must NOT be specified.Resource Quota Per PriorityClassFEATURE STATE: Kubernetes v1.17 [stable]Pods can be created at a specific priority. You can control a pod's consumption of system resources based on a pod's priority, by using the scopeSelector field in the quota spec.A quota is matched and consumed only if scopeSelector in the quota spec selects the pod.When quota is scoped for priority class using scopeSelector field, quota object is restricted to track only following resources:podscpumemoryephemeral-storagelimits.cpulimits.memorylimits.ephemeral-storagerequests.cpurequests.memoryrequests.ephemeral-storageThis example creates a quota object and matches it with pods at specific priorities. The example works as follows:Pods in the cluster have one of the three priority classes, "low", "medium", "high".One quota object is created for each priority.Save the following YAML to a file quota.yaml.policy/quota.yaml apiVersion: v1 kind: List items: - apiVersion: v1 kind: ResourceQuota metadata: name: pods-high spec: hard: cpu: "1000" memory: "200Gi" pods: "10" scopeSelector: matchExpressions: - operator: In scopeName: PriorityClass values: ["high"] - apiVersion: v1 kind: ResourceQuota metadata: name: pods-medium spec: hard: cpu: "10" memory: "20Gi" pods: "10" scopeSelector: matchExpressions: - operator: In scopeName: PriorityClass values: ["medium"] - apiVersion: v1 kind: ResourceQuota metadata: name: pods-low spec: hard: cpu: "5" memory: "10Gi" pods: "10" scopeSelector: matchExpressions: - operator: In scopeName: PriorityClass values: ["low"] Apply the YAML using kubectl create.kubectl create -f ./quota.yaml resourcequota/pods-high created resourcequota/pods-medium created resourcequota/pods-low created Verify that Used quota is 0 using kubectl describe quota.kubectl describe quota Name: pods-high Namespace: default Resource Used Hard -------- ---- ---- cpu 0 1k memory 0 200Gi pods 0 10 Name: pods-low Namespace: default Resource Used Hard -------- ---- ---- cpu 0 5 memory 0 10Gi pods 0 10 Name: pods-medium Namespace: default Resource Used Hard -------- ---- ---- cpu 0 10 memory 0 20Gi pods 0 10 Create a pod with priority "high". Save the following YAML to a file high-priority-pod.yaml.policy/high-priority-pod.yaml apiVersion: v1 kind: Pod metadata: name: high-priority spec: containers: - name: high-priority image: ubuntu command: ["/bin/sh"] args: ["-c", "while true; do echo hello; sleep 10;done"] resources: requests: memory: "10Gi" cpu: "500m" limits: memory: "10Gi" cpu: "500m" priorityClassName: high Apply it with kubectl create.kubectl create -f ./high-priority-pod.yaml Verify that "Used" stats for "high" priority quota, pods-high, has changed and that the other two quotas are unchanged.kubectl describe quota Name: pods-high Namespace: default Resource Used Hard -------- ---- ---- cpu 500m 1k memory 10Gi 200Gi pods 1 10 Name: pods-low Namespace: default Resource Used Hard -------- ---- ---- cpu 0 5 memory 0 10Gi pods 0 10 Name: pods-medium Namespace: default Resource Used Hard -------- ---- ---- cpu 0 10 memory 0 20Gi pods 0 10 Cross-namespace Pod Affinity QuotaFEATURE STATE: Kubernetes v1.24 [stable]Operators can use CrossNamespacePodAffinity quota scope to limit which namespaces are allowed to have pods with affinity terms that cross namespaces. Specifically, it controls which pods are allowed to set namespaces or namespaceSelector fields in pod affinity terms.Preventing users from using cross-namespace affinity terms might be desired since a pod with anti-affinity constraints can block pods from all other namespaces from getting scheduled in a failure domain.Using this scope operators can prevent certain namespaces (foo-ns in the example below) from having pods that use cross-namespace pod affinity by creating a resource quota object in that namespace with CrossNamespacePodAffinity scope and hard limit of 0:apiVersion: v1 kind: ResourceQuota metadata: name: disable-cross-namespace-affinity namespace: foo-ns spec: hard: pods: "0" scopeSelector: matchExpressions: - scopeName: CrossNamespacePodAffinity operator: Exists If operators want to disallow using namespaces and namespaceSelector by default, and only allow it for specific namespaces, they could configure CrossNamespacePodAffinity as a limited resource by setting the kube-apiserver flag --admission-control-config-file to the path of the following configuration file:apiVersion: apiserver.config.k8s.io/v1 kind: AdmissionConfiguration plugins: - name: "ResourceQuota" configuration: apiVersion: apiserver.config.k8s.io/v1 kind: ResourceQuotaConfiguration limitedResources: - resource: pods matchScopes: - scopeName: CrossNamespacePodAffinity operator: Exists With the above configuration, pods can use namespaces and namespaceSelector in pod affinity only if the namespace where they are created have a resource quota object with CrossNamespacePodAffinity scope and a hard limit greater than or equal to the number of pods using those fields.Resource Quota Per VolumeAttributesClassFEATURE STATE: Kubernetes v1.34 [stable] (enabled by default: true)PersistentVolumeClaims can be created with a specific volume attributes class, and might be modified after creation. You can control a PVC's consumption of storage resources based on the associated volume attributes classes, by using the scopeSelector field in the quota spec.The PVC references the associated volume attributes class by the following fields:spec.volumeAttributesClassNamestatus.currentVolumeAttributesClassNamestatus.modifyVolumeStatus.targetVolumeAttributesClassNameA quota is matched and consumed only if scopeSelector in the quota spec selects the PVC.When the quota is scoped for the volume attributes class using the scopeSelector field, the quota object is restricted to track only the following resources:persistentvolumeclaimsrequests.storageThis example creates a quota object and matches it with PVC at specific volume attributes classes. The example works as follows:PVCs in the cluster have at least one of the three volume attributes classes, "gold", "silver", "copper".One quota object is created for each volume attributes class.Save the following YAML to a file quota-vac.yaml.policy/quota-vac.yaml apiVersion: v1 kind: List items: - apiVersion: v1 kind: ResourceQuota metadata: name: pvcs-gold spec: hard: requests.storage: "10Gi" persistentvolumeclaims: "10" scopeSelector: matchExpressions: - operator: In scopeName: VolumeAttributesClass values: ["gold"] - apiVersion: v1 kind: ResourceQuota metadata: name: pvcs-silver spec: hard: requests.storage: "20Gi" persistentvolumeclaims: "10" scopeSelector: matchExpressions: - operator: In scopeName: VolumeAttributesClass values: ["silver"] - apiVersion: v1 kind: ResourceQuota metadata: name: pvcs-copper spec: hard: requests.storage: "30Gi" persistentvolumeclaims: "10" scopeSelector: matchExpressions: - operator: In scopeName: VolumeAttributesClass values: ["copper"] Apply the YAML using kubectl create.kubectl create -f ./quota-vac.yaml resourcequota/pvcs-gold created resourcequota/pvcs-silver created resourcequota/pvcs-copper created Verify that Used quota is 0 using kubectl describe quota.kubectl describe quota Name: pvcs-gold Namespace: default Resource Used Hard -------- ---- ---- persistentvolumeclaims 0 10 requests.storage 0 10Gi Name: pvcs-silver Namespace: default Resource Used Hard -------- ---- ---- persistentvolumeclaims 0 10 requests.storage 0 20Gi Name: pvcs-copper Namespace: default Resource Used Hard -------- ---- ---- persistentvolumeclaims 0 10 requests.storage 0 30Gi Create a pvc with volume attributes class "gold". Save the following YAML to a file gold-vac-pvc.yaml.policy/gold-vac-pvc.yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: gold-vac-pvc spec: accessModes: - ReadWriteOnce resources: requests: storage: 2Gi storageClassName: # change this to the name of the storage class you want to use volumeAttributesClassName: gold Apply it with kubectl create.kubectl create -f ./gold-vac-pvc.yaml Verify that "Used" stats for "gold" volume attributes class quota, pvcs-gold has changed and that the other two quotas are unchanged.kubectl describe quota Name: pvcs-gold Namespace: default Resource Used Hard -------- ---- ---- persistentvolumeclaims 1 10 requests.storage 2Gi 10Gi Name: pvcs-silver Namespace: default Resource Used Hard -------- ---- ---- persistentvolumeclaims 0 10 requests.storage 0 20Gi Name: pvcs-copper Namespace: default Resource Used Hard -------- ---- ---- persistentvolumeclaims 0 10 requests.storage 0 30Gi Once the PVC is bound, it is allowed to modify the desired volume attributes class. Let's change it to "silver" with kubectl patch.kubectl patch pvc gold-vac-pvc --type='merge' -p '{"spec":{"volumeAttributesClassName":"silver"}}' Verify that "Used" stats for "silver" volume attributes class quota, pvcs-silver has changed, pvcs-copper is unchanged, and pvcs-gold might be unchanged or released, which depends on the PVC's status.kubectl describe quota Name: pvcs-gold Namespace: default Resource Used Hard -------- ---- ---- persistentvolumeclaims 1 10 requests.storage 2Gi 10Gi Name: pvcs-silver Namespace: default Resource Used Hard -------- ---- ---- persistentvolumeclaims 1 10 requests.storage 2Gi 20Gi Name: pvcs-copper Namespace: default Resource Used Hard -------- ---- ---- persistentvolumeclaims 0 10 requests.storage 0 30Gi Let's change it to "copper" with kubectl patch.kubectl patch pvc gold-vac-pvc --type='merge' -p '{"spec":{"volumeAttributesClassName":"copper"}}' Verify that "Used" stats for "copper" volume attributes class quota, pvcs-copper has changed, pvcs-silver and pvcs-gold might be unchanged or released, which depends on the PVC's status.kubectl describe quota Name: pvcs-gold Namespace: default Resource Used Hard -------- ---- ---- persistentvolumeclaims 1 10 requests.storage 2Gi 10Gi Name: pvcs-silver Namespace: default Resource Used Hard -------- ---- ---- persistentvolumeclaims 1 10 requests.storage 2Gi 20Gi Name: pvcs-copper Namespace: default Resource Used Hard -------- ---- ---- persistentvolumeclaims 1 10 requests.storage 2Gi 30Gi Print the manifest of the PVC using the following command:kubectl get pvc gold-vac-pvc -o yaml It might show the following output:apiVersion: v1 kind: PersistentVolumeClaim metadata: name: gold-vac-pvc spec: accessModes: - ReadWriteOnce resources: requests: storage: 2Gi storageClassName: default volumeAttributesClassName: copper status: accessModes: - ReadWriteOnce capacity: storage: 2Gi currentVolumeAttributesClassName: gold phase: Bound modifyVolumeStatus: status: InProgress targetVolumeAttributesClassName: silver storageClassName: default Wait a moment for the volume modification to complete, then verify the quota again.kubectl describe quota Name: pvcs-gold Namespace: default Resource Used Hard -------- ---- ---- persistentvolumeclaims 0 10 requests.storage 0 10Gi Name: pvcs-silver Namespace: default Resource Used Hard -------- ---- ---- persistentvolumeclaims 0 10 requests.storage 0 20Gi Name: pvcs-copper Namespace: default Resource Used Hard -------- ---- ---- persistentvolumeclaims 1 10 requests.storage 2Gi 30Gi Requests compared to LimitsWhen allocating compute resources, each container may specify a request and a limit value for either CPU or memory. The quota can be configured to quota either value.If the quota has a value specified for requests.cpu or requests.memory, then it requires that every incoming container makes an explicit request for those resources. If the quota has a value specified for limits.cpu or limits.memory, then it requires that every incoming container specifies an explicit limit for those resources.Viewing and Setting Quotaskubectl supports creating, updating, and viewing quotas:kubectl create namespace myspace cat <<EOF > compute-resources.yaml apiVersion: v1 kind: ResourceQuota metadata: name: compute-resources spec: hard: requests.cpu: "1" requests.memory: "1Gi" limits.cpu: "2" limits.memory: "2Gi" requests.nvidia.com/gpu: 4 EOF kubectl create -f ./compute-resources.yaml --namespace=myspace cat <<EOF > object-counts.yaml apiVersion: v1 kind: ResourceQuota metadata: name: object-counts spec: hard: configmaps: "10" persistentvolumeclaims: "4" pods: "4" replicationcontrollers: "20" secrets: "10" services: "10" services.loadbalancers: "2" EOF kubectl create -f ./object-counts.yaml --namespace=myspace kubectl get quota --namespace=myspace NAME AGE compute-resources 30s object-counts 32s kubectl describe quota compute-resources --namespace=myspace Name: compute-resources Namespace: myspace Resource Used Hard -------- ---- ---- limits.cpu 0 2 limits.memory 0 2Gi requests.cpu 0 1 requests.memory 0 1Gi requests.nvidia.com/gpu 0 4 kubectl describe quota object-counts --namespace=myspace Name: object-counts Namespace: myspace Resource Used Hard -------- ---- ---- configmaps 0 10 persistentvolumeclaims 0 4 pods 0 4 replicationcontrollers 0 20 secrets 1 10 services 0 10 services.loadbalancers 0 2 kubectl also supports object count quota for all standard namespaced resources using the syntax count/<resource>.<group>:kubectl create namespace myspace kubectl create quota test --hard=count/deployments.apps=2,count/replicasets.apps=4,count/pods=3,count/secrets=4 --namespace=myspace kubectl create deployment nginx --image=nginx --namespace=myspace --replicas=2 kubectl describe quota --namespace=myspace Name: test Namespace: myspace Resource Used Hard -------- ---- ---- count/deployments.apps 1 2 count/pods 2 3 count/replicasets.apps 1 4 count/secrets 1 4 Quota and Cluster CapacityResourceQuotas are independent of the cluster capacity. They are expressed in absolute units. So, if you add nodes to your cluster, this does not automatically give each namespace the ability to consume more resources.Sometimes more complex policies may be desired, such as:Proportionally divide total cluster resources among several teams.Allow each tenant to grow resource usage as needed, but have a generous limit to prevent accidental resource exhaustion.Detect demand from one namespace, add nodes, and increase quota.Such policies could be implemented using ResourceQuotas as building blocks, by writing a "controller" that watches the quota usage and adjusts the quota hard limits of each namespace according to other signals.Note that resource quota divides up aggregate cluster resources, but it creates no restrictions around nodes: pods from several namespaces may run on the same node.Limit Priority Class consumption by defaultIt may be desired that pods at a particular priority, such as "cluster-services", should be allowed in a namespace, if and only if, a matching quota object exists.With this mechanism, operators are able to restrict usage of certain high priority classes to a limited number of namespaces and not every namespace will be able to consume these priority classes by default.To enforce this, kube-apiserver flag --admission-control-config-file should be used to pass path to the following configuration file:apiVersion: apiserver.config.k8s.io/v1 kind: AdmissionConfiguration plugins: - name: "ResourceQuota" configuration: apiVersion: apiserver.config.k8s.io/v1 kind: ResourceQuotaConfiguration limitedResources: - resource: pods matchScopes: - scopeName: PriorityClass operator: In values: ["cluster-services"] Then, create a resource quota object in the kube-system namespace:policy/priority-class-resourcequota.yaml apiVersion: v1 kind: ResourceQuota metadata: name: pods-cluster-services spec: scopeSelector: matchExpressions: - operator : In scopeName: PriorityClass values: ["cluster-services"]kubectl apply -f https://k8s.io/examples/policy/priority-class-resourcequota.yaml -n kube-system resourcequota/pods-cluster-services created In this case, a pod creation will be allowed if:the Pod's priorityClassName is not specified.the Pod's priorityClassName is specified to a value other than cluster-services.the Pod's priorityClassName is set to cluster-services, it is to be created in the kube-system namespace, and it has passed the resource quota check.A Pod creation request is rejected if its priorityClassName is set to cluster-services and it is to be created in a namespace other than kube-system.What's nextSee a detailed example for how to use resource quota.Read the ResourceQuota API referenceLearn about LimitRangesYou can read the historical ResourceQuota design document for more information.You can also read the Quota support for priority class design document.3 - Process ID Limits And ReservationsFEATURE STATE: Kubernetes v1.20 [stable]Kubernetes allow you to limit the number of process IDs (PIDs) that a Pod can use. You can also reserve a number of allocatable PIDs for each node for use by the operating system and daemons (rather than by Pods).Process IDs (PIDs) are a fundamental resource on nodes. It is trivial to hit the task limit without hitting any other resource limits, which can then cause instability to a host machine.Cluster administrators require mechanisms to ensure that Pods running in the cluster cannot induce PID exhaustion that prevents host daemons (such as the kubelet or kube-proxy, and potentially also the container runtime) from running. In addition, it is important to ensure that PIDs are limited among Pods in order to ensure they have limited impact on other workloads on the same node.Note:On certain Linux installations, the operating system sets the PIDs limit to a low default, such as 32768. Consider raising the value of /proc/sys/kernel/pid_max.You can configure a kubelet to limit the number of PIDs a given Pod can consume. For example, if your node's host OS is set to use a maximum of 262144 PIDs and expect to host less than 250 Pods, one can give each Pod a budget of 1000 PIDs to prevent using up that node's overall number of available PIDs. If the admin wants to overcommit PIDs similar to CPU or memory, they may do so as well with some additional risks. Either way, a single Pod will not be able to bring the whole machine down. This kind of resource limiting helps to prevent simple fork bombs from affecting operation of an entire cluster.Per-Pod PID limiting allows administrators to protect one Pod from another, but does not ensure that all Pods scheduled onto that host are unable to impact the node overall. Per-Pod limiting also does not protect the node agents themselves from PID exhaustion.You can also reserve an amount of PIDs for node overhead, separate from the allocation to Pods. This is similar to how you can reserve CPU, memory, or other resources for use by the operating system and other facilities outside of Pods and their containers.PID limiting is an important sibling to compute resource requests and limits. However, you specify it in a different way: rather than defining a Pod's resource limit in the .spec for a Pod, you configure the limit as a setting on the kubelet. Pod-defined PID limits are not currently supported.Caution:This means that the limit that applies to a Pod may be different depending on where the Pod is scheduled. To make things simple, it's easiest if all Nodes use the same PID resource limits and reservations.Node PID limitsKubernetes allows you to reserve a number of process IDs for the system use. To configure the reservation, use the parameter pid=<number> in the --system-reserved and --kube-reserved command line options to the kubelet. The value you specified declares that the specified number of process IDs will be reserved for the system as a whole and for Kubernetes system daemons respectively.Pod PID limitsKubernetes allows you to limit the number of processes running in a Pod. You specify this limit at the node level, rather than configuring it as a resource limit for a particular Pod. Each Node can have a different PID limit.To configure the limit, you can specify the command line parameter --pod-max-pids to the kubelet, or set PodPidsLimit in the kubelet configuration file.PID based evictionYou can configure kubelet to start terminating a Pod when it is misbehaving and consuming abnormal amount of resources. This feature is called eviction. You can Configure Out of Resource Handling for various eviction signals. Use pid.available eviction signal to configure the threshold for number of PIDs used by Pod. You can set soft and hard eviction policies. However, even with the hard eviction policy, if the number of PIDs growing very fast, node can still get into unstable state by hitting the node PIDs limit. Eviction signal value is calculated periodically and does NOT enforce the limit.PID limiting - per Pod and per Node sets the hard limit. Once the limit is hit, workload will start experiencing failures when trying to get a new PID. It may or may not lead to rescheduling of a Pod, depending on how workload reacts on these failures and how liveness and readiness probes are configured for the Pod. However, if limits were set correctly, you can guarantee that other Pods workload and system processes will not run out of PIDs when one Pod is misbehaving.What's nextRefer to the PID Limiting enhancement document for more information.For historical context, read Process ID Limiting for Stability Improvements in Kubernetes 1.14.Read Managing Resources for Containers.Learn how to Configure Out of Resource Handling.4 - Node Resource ManagersIn order to support latency-critical and high-throughput workloads, Kubernetes offers a suite of Resource Managers. The managers aim to co-ordinate and optimise the alignment of node's resources for pods configured with a specific requirement for CPUs, devices, and memory (hugepages) resources.Hardware topology alignment policiesTopology Manager is a kubelet component that aims to coordinate the set of components that are responsible for these optimizations. The overall resource management process is governed using the policy you specify. To learn more, read Control Topology Management Policies on a Node.Policies for assigning CPUs to PodsFEATURE STATE: Kubernetes v1.26 [stable] (enabled by default: true)Once a Pod is bound to a Node, the kubelet on that node may need to either multiplex the existing hardware (for example, sharing CPUs across multiple Pods) or allocate hardware by dedicating some resource (for example, assigning one of more CPUs for a Pod's exclusive use).By default, the kubelet uses CFS quota to enforce pod CPU limits. When the node runs many CPU-bound pods, the workload can move to different CPU cores depending on whether the pod is throttled and which CPU cores are available at scheduling time. Many workloads are not sensitive to this migration and thus work fine without any intervention.However, in workloads where CPU cache affinity and scheduling latency significantly affect workload performance, the kubelet allows alternative CPU management policies to determine some placement preferences on the node. This is implemented using the CPU Manager and its policy. There are two available policies:none: the none policy explicitly enables the existing default CPU affinity scheme, providing no affinity beyond what the OS scheduler does automatically. Limits on CPU usage for Guaranteed pods and Burstable pods are enforced using CFS quota.static: the static policy allows containers in Guaranteed pods with integer CPU requests access to exclusive CPUs on the node. This exclusivity is enforced using the cpuset cgroup controller.Note:System services such as the container runtime and the kubelet itself can continue to run on these exclusive CPUs. The exclusivity only extends to other pods.CPU Manager doesn't support offlining and onlining of CPUs at runtime.Static policyThe static policy enables finer-grained CPU management and exclusive CPU assignment. This policy manages a shared pool of CPUs that initially contains all CPUs in the node. The amount of exclusively allocatable CPUs is equal to the total number of CPUs in the node minus any CPU reservations set by the kubelet configuration. CPUs reserved by these options are taken, in integer quantity, from the initial shared pool in ascending order by physical core ID. This shared pool is the set of CPUs on which any containers in BestEffort and Burstable pods run. Containers in Guaranteed pods with fractional CPU requests also run on CPUs in the shared pool. Only containers that are part of a Guaranteed pod and have integer CPU requests are assigned exclusive CPUs.Note:The kubelet requires a CPU reservation greater than zero when the static policy is enabled. This is because a zero CPU reservation would allow the shared pool to become empty.As Guaranteed pods whose containers fit the requirements for being statically assigned are scheduled to the node, CPUs are removed from the shared pool and placed in the cpuset for the container. CFS quota is not used to bound the CPU usage of these containers as their usage is bound by the scheduling domain itself. In others words, the number of CPUs in the container cpuset is equal to the integer CPU limit specified in the pod spec. This static assignment increases CPU affinity and decreases context switches due to throttling for the CPU-bound workload.Consider the containers in the following pod specs:spec: containers: - name: nginx image: nginx The pod above runs in the BestEffort QoS class because no resource requests or limits are specified. It runs in the shared pool.spec: containers: - name: nginx image: nginx resources: limits: memory: "200Mi" requests: memory: "100Mi" The pod above runs in the Burstable QoS class because resource requests do not equal limits and the cpu quantity is not specified. It runs in the shared pool.spec: containers: - name: nginx image: nginx resources: limits: memory: "200Mi" cpu: "2" requests: memory: "100Mi" cpu: "1" The pod above runs in the Burstable QoS class because resource requests do not equal limits. It runs in the shared pool.spec: containers: - name: nginx image: nginx resources: limits: memory: "200Mi" cpu: "2" requests: memory: "200Mi" cpu: "2" The pod above runs in the Guaranteed QoS class because requests are equal to limits. And the container's resource limit for the CPU resource is an integer greater than or equal to one. The nginx container is granted 2 exclusive CPUs.spec: containers: - name: nginx image: nginx resources: limits: memory: "200Mi" cpu: "1.5" requests: memory: "200Mi" cpu: "1.5" The pod above runs in the Guaranteed QoS class because requests are equal to limits. But the container's resource limit for the CPU resource is a fraction. It runs in the shared pool.spec: containers: - name: nginx image: nginx resources: limits: memory: "200Mi" cpu: "2" The pod above runs in the Guaranteed QoS class because only limits are specified and requests are set equal to limits when not explicitly specified. And the container's resource limit for the CPU resource is an integer greater than or equal to one. The nginx container is granted 2 exclusive CPUs.Static policy optionsHere are the available policy options for the static CPU management policy, listed in alphabetical order:align-by-socket (alpha, hidden by default)Align CPUs by physical package / socket boundary, rather than logical NUMA boundaries (available since Kubernetes v1.25)distribute-cpus-across-cores (alpha, hidden by default)Allocate virtual cores, sometimes called hardware threads, across different physical cores (available since Kubernetes v1.31)distribute-cpus-across-numa (beta, visible by default)Spread CPUs across different NUMA domains, aiming for an even balance between the selected domains (available since Kubernetes v1.23)full-pcpus-only (GA, visible by default)Always allocate full physical cores (available since Kubernetes v1.22, GA since Kubernetes v1.33)strict-cpu-reservation (beta, visible by default)Prevent all the pods regardless of their Quality of Service class to run on reserved CPUs (available since Kubernetes v1.32)prefer-align-cpus-by-uncorecache (beta, visible by default)Align CPUs by uncore (Last-Level) cache boundary on a best-effort way (available since Kubernetes v1.32)You can toggle groups of options on and off based upon their maturity level using the following feature gates:CPUManagerPolicyBetaOptions (default enabled). Disable to hide beta-level options.CPUManagerPolicyAlphaOptions (default disabled). Enable to show alpha-level options.You will still have to enable each option using the cpuManagerPolicyOptions field in the kubelet configuration file.For more detail about the individual options you can configure, read on.full-pcpus-onlyIf the full-pcpus-only policy option is specified, the static policy will always allocate full physical cores. By default, without this option, the static policy allocates CPUs using a topology-aware best-fit allocation. On SMT enabled systems, the policy can allocate individual virtual cores, which correspond to hardware threads. This can lead to different containers sharing the same physical cores; this behaviour in turn contributes to the noisy neighbours problem. With the option enabled, the pod will be admitted by the kubelet only if the CPU request of all its containers can be fulfilled by allocating full physical cores. If the pod does not pass the admission, it will be put in Failed state with the message SMTAlignmentError.distribute-cpus-across-numaIf the distribute-cpus-across-numapolicy option is specified, the static policy will evenly distribute CPUs across NUMA nodes in cases where more than one NUMA node is required to satisfy the allocation. By default, the CPUManager will pack CPUs onto one NUMA node until it is filled, with any remaining CPUs simply spilling over to the next NUMA node. This can cause undesired bottlenecks in parallel code relying on barriers (and similar synchronization primitives), as this type of code tends to run only as fast as its slowest worker (which is slowed down by the fact that fewer CPUs are available on at least one NUMA node). By distributing CPUs evenly across NUMA nodes, application developers can more easily ensure that no single worker suffers from NUMA effects more than any other, improving the overall performance of these types of applications.align-by-socketIf the align-by-socket policy option is specified, CPUs will be considered aligned at the socket boundary when deciding how to allocate CPUs to a container. By default, the CPUManager aligns CPU allocations at the NUMA boundary, which could result in performance degradation if CPUs need to be pulled from more than one NUMA node to satisfy the allocation. Although it tries to ensure that all CPUs are allocated from the minimum number of NUMA nodes, there is no guarantee that those NUMA nodes will be on the same socket. By directing the CPUManager to explicitly align CPUs at the socket boundary rather than the NUMA boundary, we are able to avoid such issues. Note, this policy option is not compatible with TopologyManager single-numa-node policy and does not apply to hardware where the number of sockets is greater than number of NUMA nodes.distribute-cpus-across-coresIf the distribute-cpus-across-cores policy option is specified, the static policy will attempt to allocate virtual cores (hardware threads) across different physical cores. By default, the CPUManager tends to pack CPUs onto as few physical cores as possible, which can lead to contention among CPUs on the same physical core and result in performance bottlenecks. By enabling the distribute-cpus-across-cores policy, the static policy ensures that CPUs are distributed across as many physical cores as possible, reducing the contention on the same physical core and thereby improving overall performance. However, it is important to note that this strategy might be less effective when the system is heavily loaded. Under such conditions, the benefit of reducing contention diminishes. Conversely, default behavior can help in reducing inter-core communication overhead, potentially providing better performance under high load conditions.strict-cpu-reservationThe reservedSystemCPUs parameter in KubeletConfiguration, or the deprecated kubelet command line option --reserved-cpus, defines an explicit CPU set for OS system daemons and kubernetes system daemons. More details of this parameter can be found on the Explicitly Reserved CPU List page. By default, this isolation is implemented only for guaranteed pods with integer CPU requests not for burstable and best-effort pods (and guaranteed pods with fractional CPU requests). Admission is only comparing the CPU requests against the allocatable CPUs. Since the CPU limit is higher than the request, the default behaviour allows burstable and best-effort pods to use up the capacity of reservedSystemCPUs and cause host OS services to starve in real life deployments. If the strict-cpu-reservation policy option is enabled, the static policy will not allow any workload to use the CPU cores specified in reservedSystemCPUs.prefer-align-cpus-by-uncorecacheIf the prefer-align-cpus-by-uncorecache policy is specified, the static policy will allocate CPU resources for individual containers such that all CPUs assigned to a container share the same uncore cache block (also known as the Last-Level Cache or LLC). By default, the CPUManager will tightly pack CPU assignments which can result in containers being assigned CPUs from multiple uncore caches. This option enables the CPUManager to allocate CPUs in a way that maximizes the efficient use of the uncore cache. Allocation is performed on a best-effort basis, aiming to affine as many CPUs as possible within the same uncore cache. If the container's CPU requirement exceeds the CPU capacity of a single uncore cache, the CPUManager minimizes the number of uncore caches used in order to maintain optimal uncore cache alignment. Specific workloads can benefit in performance from the reduction of inter-cache latency and noisy neighbors at the cache level. If the CPUManager cannot align optimally while the node has sufficient resources, the container will still be admitted using the default packed behavior.Memory Management PoliciesFEATURE STATE: Kubernetes v1.32 [stable] (enabled by default: true)The Kubernetes Memory Manager enables the feature of guaranteed memory (and hugepages) allocation for pods in the Guaranteed QoS class.The Memory Manager employs hint generation protocol to yield the most suitable NUMA affinity for a pod. The Memory Manager feeds the central manager (Topology Manager) with these affinity hints. Based on both the hints and Topology Manager policy, the pod is rejected or admitted to the node.Moreover, the Memory Manager ensures that the memory which a pod requests is allocated from a minimum number of NUMA nodes.Other resource managersThe configuration of individual managers is elaborated in dedicated documents:Device Manager


# kubectl create rolebinding

kubectl create rolebindingSynopsisCreate a role binding for a particular role or cluster role.kubectl create rolebinding NAME --clusterrole=NAME|--role=NAME [--user=username] [--group=groupname] [--serviceaccount=namespace:serviceaccountname] [--dry-run=server|client|none] Examples # Create a role binding for user1, user2, and group1 using the admin cluster role kubectl create rolebinding admin --clusterrole=admin --user=user1 --user=user2 --group=group1 # Create a role binding for service account monitoring:sa-dev using the admin role kubectl create rolebinding admin-binding --role=admin --serviceaccount=monitoring:sa-dev Options--allow-missing-template-keys Default: trueIf true, ignore any errors in templates when a field or map key is missing in the template. Only applies to golang and jsonpath output formats.--clusterrole stringClusterRole this RoleBinding should reference--dry-run string[="unchanged"] Default: "none"Must be "none", "server", or "client". If client strategy, only print the object that would be sent, without sending it. If server strategy, submit server-side request without persisting the resource.--field-manager string Default: "kubectl-create"Name of the manager used to track field ownership.--group stringsGroups to bind to the role. The flag can be repeated to add multiple groups.-h, --helphelp for rolebinding-o, --output stringOutput format. One of: (json, yaml, name, go-template, go-template-file, template, templatefile, jsonpath, jsonpath-as-json, jsonpath-file).--role stringRole this RoleBinding should reference--save-configIf true, the configuration of current object will be saved in its annotation. Otherwise, the annotation will be unchanged. This flag is useful when you want to perform kubectl apply on this object in the future.--serviceaccount stringsService accounts to bind to the role, in the format <namespace>:<name>. The flag can be repeated to add multiple service accounts.--show-managed-fieldsIf true, keep the managedFields when printing objects in JSON or YAML format.--template stringTemplate string or path to template file to use when -o=go-template, -o=go-template-file. The template format is golang templates [http://golang.org/pkg/text/template/#pkg-overview].--user stringsUsernames to bind to the role. The flag can be repeated to add multiple users.--validate string[="strict"] Default: "strict"Must be one of: strict (or true), warn, ignore (or false). "true" or "strict" will use a schema to validate the input and fail the request if invalid. It will perform server side validation if ServerSideFieldValidation is enabled on the api-server, but will fall back to less reliable client-side validation if not. "warn" will warn about unknown or duplicate fields without blocking the request if server-side field validation is enabled on the API server, and behave as "ignore" otherwise. "false" or "ignore" will not perform any schema validation, silently dropping any unknown or duplicate fields.Parent Options Inherited--as stringUsername to impersonate for the operation. User could be a regular user or a service account in a namespace.--as-group stringsGroup to impersonate for the operation, this flag can be repeated to specify multiple groups.--as-uid stringUID to impersonate for the operation.--cache-dir string Default: "$HOME/.kube/cache"Default cache directory--certificate-authority stringPath to a cert file for the certificate authority--client-certificate stringPath to a client certificate file for TLS--client-key stringPath to a client key file for TLS--cluster stringThe name of the kubeconfig cluster to use--context stringThe name of the kubeconfig context to use--disable-compressionIf true, opt-out of response compression for all requests to the server--insecure-skip-tls-verifyIf true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure--kubeconfig stringPath to the kubeconfig file to use for CLI requests.--kuberc stringPath to the kuberc file to use for preferences. This can be disabled by exporting KUBECTL_KUBERC=false feature gate or turning off the feature KUBERC=off.--match-server-versionRequire server version to match client version-n, --namespace stringIf present, the namespace scope for this CLI request--password stringPassword for basic authentication to the API server--profile string Default: "none"Name of profile to capture. One of (none|cpu|heap|goroutine|threadcreate|block|mutex)--profile-output string Default: "profile.pprof"Name of the file to write the profile to--request-timeout string Default: "0"The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests.-s, --server stringThe address and port of the Kubernetes API server--storage-driver-buffer-duration duration Default: 1m0sWrites in the storage driver will be buffered for this duration, and committed to the non memory backends as a single transaction--storage-driver-db string Default: "cadvisor"database name--storage-driver-host string Default: "localhost:8086"database host:port--storage-driver-password string Default: "root"database password--storage-driver-secureuse secure connection with database--storage-driver-table string Default: "stats"table name--storage-driver-user string Default: "root"database username--tls-server-name stringServer name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used--token stringBearer token for authentication to the API server--username stringUsername for basic authentication to the API server--version version[=true]--version, --version=raw prints version information and quits; --version=vX.Y.Z... sets the reported version--warnings-as-errorsTreat warnings received from the server as errors and exit with a non-zero exit codeSee Alsokubectl create - Create a resource from a file or from stdinThis page is automatically generated.If you plan to report an issue with this page, mention that the page is auto-generated in your issue description. The fix may need to happen elsewhere in the Kubernetes project.


# kubectl create

kubectl createSynopsisCreate a resource from a file or from stdin.JSON and YAML formats are accepted.kubectl create -f FILENAME Examples # Create a pod using the data in pod.json kubectl create -f ./pod.json # Create a pod based on the JSON passed into stdin cat pod.json | kubectl create -f - # Edit the data in registry.yaml in JSON then create the resource using the edited data kubectl create -f registry.yaml --edit -o json Options--allow-missing-template-keys Default: trueIf true, ignore any errors in templates when a field or map key is missing in the template. Only applies to golang and jsonpath output formats.--dry-run string[="unchanged"] Default: "none"Must be "none", "server", or "client". If client strategy, only print the object that would be sent, without sending it. If server strategy, submit server-side request without persisting the resource.--editEdit the API resource before creating--field-manager string Default: "kubectl-create"Name of the manager used to track field ownership.-f, --filename stringsFilename, directory, or URL to files to use to create the resource-h, --helphelp for create-k, --kustomize stringProcess the kustomization directory. This flag can't be used together with -f or -R.-o, --output stringOutput format. One of: (json, yaml, name, go-template, go-template-file, template, templatefile, jsonpath, jsonpath-as-json, jsonpath-file).--raw stringRaw URI to POST to the server. Uses the transport specified by the kubeconfig file.-R, --recursiveProcess the directory used in -f, --filename recursively. Useful when you want to manage related manifests organized within the same directory.--save-configIf true, the configuration of current object will be saved in its annotation. Otherwise, the annotation will be unchanged. This flag is useful when you want to perform kubectl apply on this object in the future.-l, --selector stringSelector (label query) to filter on, supports '=', '==', '!=', 'in', 'notin'.(e.g. -l key1=value1,key2=value2,key3 in (value3)). Matching objects must satisfy all of the specified label constraints.--show-managed-fieldsIf true, keep the managedFields when printing objects in JSON or YAML format.--template stringTemplate string or path to template file to use when -o=go-template, -o=go-template-file. The template format is golang templates [http://golang.org/pkg/text/template/#pkg-overview].--validate string[="strict"] Default: "strict"Must be one of: strict (or true), warn, ignore (or false). "true" or "strict" will use a schema to validate the input and fail the request if invalid. It will perform server side validation if ServerSideFieldValidation is enabled on the api-server, but will fall back to less reliable client-side validation if not. "warn" will warn about unknown or duplicate fields without blocking the request if server-side field validation is enabled on the API server, and behave as "ignore" otherwise. "false" or "ignore" will not perform any schema validation, silently dropping any unknown or duplicate fields.--windows-line-endingsOnly relevant if --edit=true. Defaults to the line ending native to your platform.Parent Options Inherited--as stringUsername to impersonate for the operation. User could be a regular user or a service account in a namespace.--as-group stringsGroup to impersonate for the operation, this flag can be repeated to specify multiple groups.--as-uid stringUID to impersonate for the operation.--cache-dir string Default: "$HOME/.kube/cache"Default cache directory--certificate-authority stringPath to a cert file for the certificate authority--client-certificate stringPath to a client certificate file for TLS--client-key stringPath to a client key file for TLS--cluster stringThe name of the kubeconfig cluster to use--context stringThe name of the kubeconfig context to use--disable-compressionIf true, opt-out of response compression for all requests to the server--insecure-skip-tls-verifyIf true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure--kubeconfig stringPath to the kubeconfig file to use for CLI requests.--kuberc stringPath to the kuberc file to use for preferences. This can be disabled by exporting KUBECTL_KUBERC=false feature gate or turning off the feature KUBERC=off.--match-server-versionRequire server version to match client version-n, --namespace stringIf present, the namespace scope for this CLI request--password stringPassword for basic authentication to the API server--profile string Default: "none"Name of profile to capture. One of (none|cpu|heap|goroutine|threadcreate|block|mutex)--profile-output string Default: "profile.pprof"Name of the file to write the profile to--request-timeout string Default: "0"The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests.-s, --server stringThe address and port of the Kubernetes API server--storage-driver-buffer-duration duration Default: 1m0sWrites in the storage driver will be buffered for this duration, and committed to the non memory backends as a single transaction--storage-driver-db string Default: "cadvisor"database name--storage-driver-host string Default: "localhost:8086"database host:port--storage-driver-password string Default: "root"database password--storage-driver-secureuse secure connection with database--storage-driver-table string Default: "stats"table name--storage-driver-user string Default: "root"database username--tls-server-name stringServer name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used--token stringBearer token for authentication to the API server--user stringThe name of the kubeconfig user to use--username stringUsername for basic authentication to the API server--version version[=true]--version, --version=raw prints version information and quits; --version=vX.Y.Z... sets the reported version--warnings-as-errorsTreat warnings received from the server as errors and exit with a non-zero exit codeSee Alsokubectl - kubectl controls the Kubernetes cluster managerkubectl create clusterrole - Create a cluster rolekubectl create clusterrolebinding - Create a cluster role binding for a particular cluster rolekubectl create configmap - Create a config map from a local file, directory or literal valuekubectl create cronjob - Create a cron job with the specified namekubectl create deployment - Create a deployment with the specified namekubectl create ingress - Create an ingress with the specified namekubectl create job - Create a job with the specified namekubectl create namespace - Create a namespace with the specified namekubectl create poddisruptionbudget - Create a pod disruption budget with the specified namekubectl create priorityclass - Create a priority class with the specified namekubectl create quota - Create a quota with the specified namekubectl create role - Create a role with single rulekubectl create rolebinding - Create a role binding for a particular role or cluster rolekubectl create secret - Create a secret using a specified subcommandkubectl create service - Create a service using a specified subcommandkubectl create serviceaccount - Create a service account with the specified namekubectl create token - Request a service account tokenThis page is automatically generated.If you plan to report an issue with this page, mention that the page is auto-generated in your issue description. The fix may need to happen elsewhere in the Kubernetes project.


# kubectl rollout

kubectl rolloutSynopsisManage the rollout of one or many resources.Valid resource types include:deploymentsdaemonsetsstatefulsetskubectl rollout SUBCOMMAND Examples # Rollback to the previous deployment kubectl rollout undo deployment/abc # Check the rollout status of a daemonset kubectl rollout status daemonset/foo # Restart a deployment kubectl rollout restart deployment/abc # Restart deployments with the 'app=nginx' label kubectl rollout restart deployment --selector=app=nginx Options-h, --helphelp for rolloutParent Options Inherited--as stringUsername to impersonate for the operation. User could be a regular user or a service account in a namespace.--as-group stringsGroup to impersonate for the operation, this flag can be repeated to specify multiple groups.--as-uid stringUID to impersonate for the operation.--cache-dir string Default: "$HOME/.kube/cache"Default cache directory--certificate-authority stringPath to a cert file for the certificate authority--client-certificate stringPath to a client certificate file for TLS--client-key stringPath to a client key file for TLS--cluster stringThe name of the kubeconfig cluster to use--context stringThe name of the kubeconfig context to use--disable-compressionIf true, opt-out of response compression for all requests to the server--insecure-skip-tls-verifyIf true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure--kubeconfig stringPath to the kubeconfig file to use for CLI requests.--kuberc stringPath to the kuberc file to use for preferences. This can be disabled by exporting KUBECTL_KUBERC=false feature gate or turning off the feature KUBERC=off.--match-server-versionRequire server version to match client version-n, --namespace stringIf present, the namespace scope for this CLI request--password stringPassword for basic authentication to the API server--profile string Default: "none"Name of profile to capture. One of (none|cpu|heap|goroutine|threadcreate|block|mutex)--profile-output string Default: "profile.pprof"Name of the file to write the profile to--request-timeout string Default: "0"The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests.-s, --server stringThe address and port of the Kubernetes API server--storage-driver-buffer-duration duration Default: 1m0sWrites in the storage driver will be buffered for this duration, and committed to the non memory backends as a single transaction--storage-driver-db string Default: "cadvisor"database name--storage-driver-host string Default: "localhost:8086"database host:port--storage-driver-password string Default: "root"database password--storage-driver-secureuse secure connection with database--storage-driver-table string Default: "stats"table name--storage-driver-user string Default: "root"database username--tls-server-name stringServer name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used--token stringBearer token for authentication to the API server--user stringThe name of the kubeconfig user to use--username stringUsername for basic authentication to the API server--version version[=true]--version, --version=raw prints version information and quits; --version=vX.Y.Z... sets the reported version--warnings-as-errorsTreat warnings received from the server as errors and exit with a non-zero exit codeSee Alsokubectl - kubectl controls the Kubernetes cluster managerkubectl rollout history - View rollout historykubectl rollout pause - Mark the provided resource as pausedkubectl rollout restart - Restart a resourcekubectl rollout resume - Resume a paused resourcekubectl rollout status - Show the status of the rolloutkubectl rollout undo - Undo a previous rolloutThis page is automatically generated.If you plan to report an issue with this page, mention that the page is auto-generated in your issue description. The fix may need to happen elsewhere in the Kubernetes project.


# Define a Command and Arguments for a Container

Define a Command and Arguments for a ContainerThis page shows how to define commands and arguments when you run a container in a Pod.Before you beginYou need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. It is recommended to run this tutorial on a cluster with at least two nodes that are not acting as control plane hosts. If you do not already have a cluster, you can create one by using minikube or you can use one of these Kubernetes playgrounds:iximiuz LabsKillercodaKodeKloudPlay with KubernetesTo check the version, enter kubectl version.Define a command and arguments when you create a PodWhen you create a Pod, you can define a command and arguments for the containers that run in the Pod. To define a command, include the command field in the configuration file. To define arguments for the command, include the args field in the configuration file. The command and arguments that you define cannot be changed after the Pod is created.The command and arguments that you define in the configuration file override the default command and arguments provided by the container image. If you define args, but do not define a command, the default command is used with your new arguments.Note:The command field corresponds to ENTRYPOINT, and the args field corresponds to CMD in some container runtimes.In this exercise, you create a Pod that runs one container. The configuration file for the Pod defines a command and two arguments:pods/commands.yaml apiVersion: v1 kind: Pod metadata: name: command-demo labels: purpose: demonstrate-command spec: containers: - name: command-demo-container image: debian command: ["printenv"] args: ["HOSTNAME", "KUBERNETES_PORT"] restartPolicy: OnFailure Create a Pod based on the YAML configuration file:kubectl apply -f https://k8s.io/examples/pods/commands.yaml List the running Pods:kubectl get pods The output shows that the container that ran in the command-demo Pod has completed.To see the output of the command that ran in the container, view the logs from the Pod:kubectl logs command-demo The output shows the values of the HOSTNAME and KUBERNETES_PORT environment variables:command-demo tcp://10.3.240.1:443 Use environment variables to define argumentsIn the preceding example, you defined the arguments directly by providing strings. As an alternative to providing strings directly, you can define arguments by using environment variables:env: - name: MESSAGE value: "hello world" command: ["/bin/echo"] args: ["$(MESSAGE)"] This means you can define an argument for a Pod using any of the techniques available for defining environment variables, including ConfigMaps and Secrets.Note:The environment variable appears in parentheses, "$(VAR)". This is required for the variable to be expanded in the command or args field.Run a command in a shellIn some cases, you need your command to run in a shell. For example, your command might consist of several commands piped together, or it might be a shell script. To run your command in a shell, wrap it like this:command: ["/bin/sh"] args: ["-c", "while true; do echo hello; sleep 10;done"] What's nextLearn more about configuring pods and containers.Learn more about running commands in a container.See Container.


# Perform a Rollback on a DaemonSet

Perform a Rollback on a DaemonSetThis page shows how to perform a rollback on a DaemonSet.Before you beginYou need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. It is recommended to run this tutorial on a cluster with at least two nodes that are not acting as control plane hosts. If you do not already have a cluster, you can create one by using minikube or you can use one of these Kubernetes playgrounds:iximiuz LabsKillercodaKodeKloudPlay with KubernetesYour Kubernetes server must be at or later than version 1.7.To check the version, enter kubectl version.You should already know how to perform a rolling update on a DaemonSet.Performing a rollback on a DaemonSetStep 1: Find the DaemonSet revision you want to roll back toYou can skip this step if you only want to roll back to the last revision.List all revisions of a DaemonSet:kubectl rollout history daemonset <daemonset-name> This returns a list of DaemonSet revisions:daemonsets "<daemonset-name>" REVISION CHANGE-CAUSE 1 ... 2 ... ... Change cause is copied from DaemonSet annotation kubernetes.io/change-cause to its revisions upon creation. You may specify --record=true in kubectl to record the command executed in the change cause annotation.To see the details of a specific revision:kubectl rollout history daemonset <daemonset-name> --revision=1 This returns the details of that revision:daemonsets "<daemonset-name>" with revision #1 Pod Template: Labels: foo=bar Containers: app: Image: ... Port: ... Environment: ... Mounts: ... Volumes: ... Step 2: Roll back to a specific revision# Specify the revision number you get from Step 1 in --to-revision kubectl rollout undo daemonset <daemonset-name> --to-revision=<revision> If it succeeds, the command returns:daemonset "<daemonset-name>" rolled back Note:If --to-revision flag is not specified, kubectl picks the most recent revision.Step 3: Watch the progress of the DaemonSet rollbackkubectl rollout undo daemonset tells the server to start rolling back the DaemonSet. The real rollback is done asynchronously inside the cluster control plane.To watch the progress of the rollback:kubectl rollout status ds/<daemonset-name> When the rollback is complete, the output is similar to:daemonset "<daemonset-name>" successfully rolled out Understanding DaemonSet revisionsIn the previous kubectl rollout history step, you got a list of DaemonSet revisions. Each revision is stored in a resource named ControllerRevision.To see what is stored in each revision, find the DaemonSet revision raw resources:kubectl get controllerrevision -l <daemonset-selector-key>=<daemonset-selector-value> This returns a list of ControllerRevisions:NAME CONTROLLER REVISION AGE <daemonset-name>-<revision-hash> DaemonSet/<daemonset-name> 1 1h <daemonset-name>-<revision-hash> DaemonSet/<daemonset-name> 2 1h Each ControllerRevision stores the annotations and template of a DaemonSet revision.kubectl rollout undo takes a specific ControllerRevision and replaces DaemonSet template with the template stored in the ControllerRevision. kubectl rollout undo is equivalent to updating DaemonSet template to a previous revision through other commands, such as kubectl edit or kubectl apply.Note:DaemonSet revisions only roll forward. That is to say, after a rollback completes, the revision number (.revision field) of the ControllerRevision being rolled back to will advance. For example, if you have revision 1 and 2 in the system, and roll back from revision 2 to revision 1, the ControllerRevision with .revision: 1 will become .revision: 3.TroubleshootingSee troubleshooting DaemonSet rolling update.