# ReplicaSet

ReplicaSetReplicaSet ensures that a specified number of pod replicas are running at any given time.apiVersion: apps/v1import "k8s.io/api/apps/v1"ReplicaSetReplicaSet ensures that a specified number of pod replicas are running at any given time.apiVersion: apps/v1kind: ReplicaSetmetadata (ObjectMeta)If the Labels of a ReplicaSet are empty, they are defaulted to be the same as the Pod(s) that the ReplicaSet manages. Standard object's metadata. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadataspec (ReplicaSetSpec)Spec defines the specification of the desired behavior of the ReplicaSet. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-statusstatus (ReplicaSetStatus)Status is the most recently observed status of the ReplicaSet. This data may be out of date by some window of time. Populated by the system. Read-only. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-statusReplicaSetSpecReplicaSetSpec is the specification of a ReplicaSet.selector (LabelSelector), requiredSelector is a label query over pods that should match the replica count. Label keys and values that must match in order to be controlled by this replica set. It must match the pod template's labels. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#label-selectorstemplate (PodTemplateSpec)Template is the object that describes the pod that will be created if insufficient replicas are detected. More info: https://kubernetes.io/docs/concepts/workloads/controllers/replicaset/#pod-templatereplicas (int32)Replicas is the number of desired pods. This is a pointer to distinguish between explicit zero and unspecified. Defaults to 1. More info: https://kubernetes.io/docs/concepts/workloads/controllers/replicasetminReadySeconds (int32)Minimum number of seconds for which a newly created pod should be ready without any of its container crashing, for it to be considered available. Defaults to 0 (pod will be considered available as soon as it is ready)ReplicaSetStatusReplicaSetStatus represents the current status of a ReplicaSet.replicas (int32), requiredReplicas is the most recently observed number of non-terminating pods. More info: https://kubernetes.io/docs/concepts/workloads/controllers/replicasetavailableReplicas (int32)The number of available non-terminating pods (ready for at least minReadySeconds) for this replica set.readyReplicas (int32)The number of non-terminating pods targeted by this ReplicaSet with a Ready Condition.terminatingReplicas (int32)The number of terminating pods for this replica set. Terminating pods have a non-null .metadata.deletionTimestamp and have not yet reached the Failed or Succeeded .status.phase.This is an alpha field. Enable DeploymentReplicaSetTerminatingReplicas to be able to use this field.fullyLabeledReplicas (int32)The number of non-terminating pods that have labels matching the labels of the pod template of the replicaset.conditions ([]ReplicaSetCondition)Patch strategy: merge on key typeMap: unique values on key type will be kept during a mergeRepresents the latest available observations of a replica set's current state.ReplicaSetCondition describes the state of a replica set at a certain point.conditions.status (string), requiredStatus of the condition, one of True, False, Unknown.conditions.type (string), requiredType of replica set condition.conditions.lastTransitionTime (Time)The last time the condition transitioned from one status to another.Time is a wrapper around time.Time which supports correct marshaling to YAML and JSON. Wrappers are provided for many of the factory methods that the time package offers.conditions.message (string)A human readable message indicating details about the transition.conditions.reason (string)The reason for the condition's last transition.observedGeneration (int64)ObservedGeneration reflects the generation of the most recently observed ReplicaSet.ReplicaSetListReplicaSetList is a collection of ReplicaSets.apiVersion: apps/v1kind: ReplicaSetListmetadata (ListMeta)Standard list metadata. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kindsitems ([]ReplicaSet), requiredList of ReplicaSets. More info: https://kubernetes.io/docs/concepts/workloads/controllers/replicasetOperationsget read the specified ReplicaSetHTTP RequestGET /apis/apps/v1/namespaces/{namespace}/replicasets/{name}Parametersname (in path): string, requiredname of the ReplicaSetnamespace (in path): string, requirednamespacepretty (in query): stringprettyResponse200 (ReplicaSet): OK401: Unauthorizedget read status of the specified ReplicaSetHTTP RequestGET /apis/apps/v1/namespaces/{namespace}/replicasets/{name}/statusParametersname (in path): string, requiredname of the ReplicaSetnamespace (in path): string, requirednamespacepretty (in query): stringprettyResponse200 (ReplicaSet): OK401: Unauthorizedlist list or watch objects of kind ReplicaSetHTTP RequestGET /apis/apps/v1/namespaces/{namespace}/replicasetsParametersnamespace (in path): string, requirednamespaceallowWatchBookmarks (in query): booleanallowWatchBookmarkscontinue (in query): stringcontinuefieldSelector (in query): stringfieldSelectorlabelSelector (in query): stringlabelSelectorlimit (in query): integerlimitpretty (in query): stringprettyresourceVersion (in query): stringresourceVersionresourceVersionMatch (in query): stringresourceVersionMatchsendInitialEvents (in query): booleansendInitialEventstimeoutSeconds (in query): integertimeoutSecondswatch (in query): booleanwatchResponse200 (ReplicaSetList): OK401: Unauthorizedlist list or watch objects of kind ReplicaSetHTTP RequestGET /apis/apps/v1/replicasetsParametersallowWatchBookmarks (in query): booleanallowWatchBookmarkscontinue (in query): stringcontinuefieldSelector (in query): stringfieldSelectorlabelSelector (in query): stringlabelSelectorlimit (in query): integerlimitpretty (in query): stringprettyresourceVersion (in query): stringresourceVersionresourceVersionMatch (in query): stringresourceVersionMatchsendInitialEvents (in query): booleansendInitialEventstimeoutSeconds (in query): integertimeoutSecondswatch (in query): booleanwatchResponse200 (ReplicaSetList): OK401: Unauthorizedcreate create a ReplicaSetHTTP RequestPOST /apis/apps/v1/namespaces/{namespace}/replicasetsParametersnamespace (in path): string, requirednamespacebody: ReplicaSet, requireddryRun (in query): stringdryRunfieldManager (in query): stringfieldManagerfieldValidation (in query): stringfieldValidationpretty (in query): stringprettyResponse200 (ReplicaSet): OK201 (ReplicaSet): Created202 (ReplicaSet): Accepted401: Unauthorizedupdate replace the specified ReplicaSetHTTP RequestPUT /apis/apps/v1/namespaces/{namespace}/replicasets/{name}Parametersname (in path): string, requiredname of the ReplicaSetnamespace (in path): string, requirednamespacebody: ReplicaSet, requireddryRun (in query): stringdryRunfieldManager (in query): stringfieldManagerfieldValidation (in query): stringfieldValidationpretty (in query): stringprettyResponse200 (ReplicaSet): OK201 (ReplicaSet): Created401: Unauthorizedupdate replace status of the specified ReplicaSetHTTP RequestPUT /apis/apps/v1/namespaces/{namespace}/replicasets/{name}/statusParametersname (in path): string, requiredname of the ReplicaSetnamespace (in path): string, requirednamespacebody: ReplicaSet, requireddryRun (in query): stringdryRunfieldManager (in query): stringfieldManagerfieldValidation (in query): stringfieldValidationpretty (in query): stringprettyResponse200 (ReplicaSet): OK201 (ReplicaSet): Created401: Unauthorizedpatch partially update the specified ReplicaSetHTTP RequestPATCH /apis/apps/v1/namespaces/{namespace}/replicasets/{name}Parametersname (in path): string, requiredname of the ReplicaSetnamespace (in path): string, requirednamespacebody: Patch, requireddryRun (in query): stringdryRunfieldManager (in query): stringfieldManagerfieldValidation (in query): stringfieldValidationforce (in query): booleanforcepretty (in query): stringprettyResponse200 (ReplicaSet): OK201 (ReplicaSet): Created401: Unauthorizedpatch partially update status of the specified ReplicaSetHTTP RequestPATCH /apis/apps/v1/namespaces/{namespace}/replicasets/{name}/statusParametersname (in path): string, requiredname of the ReplicaSetnamespace (in path): string, requirednamespacebody: Patch, requireddryRun (in query): stringdryRunfieldManager (in query): stringfieldManagerfieldValidation (in query): stringfieldValidationforce (in query): booleanforcepretty (in query): stringprettyResponse200 (ReplicaSet): OK201 (ReplicaSet): Created401: Unauthorizeddelete delete a ReplicaSetHTTP RequestDELETE /apis/apps/v1/namespaces/{namespace}/replicasets/{name}Parametersname (in path): string, requiredname of the ReplicaSetnamespace (in path): string, requirednamespacebody: DeleteOptionsdryRun (in query): stringdryRungracePeriodSeconds (in query): integergracePeriodSecondsignoreStoreReadErrorWithClusterBreakingPotential (in query): booleanignoreStoreReadErrorWithClusterBreakingPotentialpretty (in query): stringprettypropagationPolicy (in query): stringpropagationPolicyResponse200 (Status): OK202 (Status): Accepted401: Unauthorizeddeletecollection delete collection of ReplicaSetHTTP RequestDELETE /apis/apps/v1/namespaces/{namespace}/replicasetsParametersnamespace (in path): string, requirednamespacebody: DeleteOptionscontinue (in query): stringcontinuedryRun (in query): stringdryRunfieldSelector (in query): stringfieldSelectorgracePeriodSeconds (in query): integergracePeriodSecondsignoreStoreReadErrorWithClusterBreakingPotential (in query): booleanignoreStoreReadErrorWithClusterBreakingPotentiallabelSelector (in query): stringlabelSelectorlimit (in query): integerlimitpretty (in query): stringprettypropagationPolicy (in query): stringpropagationPolicyresourceVersion (in query): stringresourceVersionresourceVersionMatch (in query): stringresourceVersionMatchsendInitialEvents (in query): booleansendInitialEventstimeoutSeconds (in query): integertimeoutSecondsResponse200 (Status): OK401: UnauthorizedThis page is automatically generated.If you plan to report an issue with this page, mention that the page is auto-generated in your issue description. The fix may need to happen elsewhere in the Kubernetes project.


# Labels and Selectors

Labels and SelectorsLabels are key/value pairs that are attached to objects such as Pods. Labels are intended to be used to specify identifying attributes of objects that are meaningful and relevant to users, but do not directly imply semantics to the core system. Labels can be used to organize and to select subsets of objects. Labels can be attached to objects at creation time and subsequently added and modified at any time. Each object can have a set of key/value labels defined. Each Key must be unique for a given object."metadata": { "labels": { "key1" : "value1", "key2" : "value2" } } Labels allow for efficient queries and watches and are ideal for use in UIs and CLIs. Non-identifying information should be recorded using annotations.MotivationLabels enable users to map their own organizational structures onto system objects in a loosely coupled fashion, without requiring clients to store these mappings.Service deployments and batch processing pipelines are often multi-dimensional entities (e.g., multiple partitions or deployments, multiple release tracks, multiple tiers, multiple micro-services per tier). Management often requires cross-cutting operations, which breaks encapsulation of strictly hierarchical representations, especially rigid hierarchies determined by the infrastructure rather than by users.Example labels:"release" : "stable", "release" : "canary""environment" : "dev", "environment" : "qa", "environment" : "production""tier" : "frontend", "tier" : "backend", "tier" : "cache""partition" : "customerA", "partition" : "customerB""track" : "daily", "track" : "weekly"These are examples of commonly used labels; you are free to develop your own conventions. Keep in mind that label Key must be unique for a given object.Syntax and character setLabels are key/value pairs. Valid label keys have two segments: an optional prefix and name, separated by a slash (/). The name segment is required and must be 63 characters or less, beginning and ending with an alphanumeric character ([a-z0-9A-Z]) with dashes (-), underscores (_), dots (.), and alphanumerics between. The prefix is optional. If specified, the prefix must be a DNS subdomain: a series of DNS labels separated by dots (.), not longer than 253 characters in total, followed by a slash (/).If the prefix is omitted, the label Key is presumed to be private to the user. Automated system components (e.g. kube-scheduler, kube-controller-manager, kube-apiserver, kubectl, or other third-party automation) which add labels to end-user objects must specify a prefix.The kubernetes.io/ and k8s.io/ prefixes are reserved for Kubernetes core components.Valid label value:must be 63 characters or less (can be empty),unless empty, must begin and end with an alphanumeric character ([a-z0-9A-Z]),could contain dashes (-), underscores (_), dots (.), and alphanumerics between.For example, here's a manifest for a Pod that has two labels environment: production and app: nginx:apiVersion: v1 kind: Pod metadata: name: label-demo labels: environment: production app: nginx spec: containers: - name: nginx image: nginx:1.14.2 ports: - containerPort: 80 Label selectorsUnlike names and UIDs, labels do not provide uniqueness. In general, we expect many objects to carry the same label(s).Via a label selector, the client/user can identify a set of objects. The label selector is the core grouping primitive in Kubernetes.The API currently supports two types of selectors: equality-based and set-based. A label selector can be made of multiple requirements which are comma-separated. In the case of multiple requirements, all must be satisfied so the comma separator acts as a logical AND (&&) operator.The semantics of empty or non-specified selectors are dependent on the context, and API types that use selectors should document the validity and meaning of them.Note:For some API types, such as ReplicaSets, the label selectors of two instances must not overlap within a namespace, or the controller can see that as conflicting instructions and fail to determine how many replicas should be present.Caution:For both equality-based and set-based conditions there is no logical OR (||) operator. Ensure your filter statements are structured accordingly.Equality-based requirementEquality- or inequality-based requirements allow filtering by label keys and values. Matching objects must satisfy all of the specified label constraints, though they may have additional labels as well. Three kinds of operators are admitted =,==,!=. The first two represent equality (and are synonyms), while the latter represents inequality. For example:environment = production tier != frontend The former selects all resources with key equal to environment and value equal to production. The latter selects all resources with key equal to tier and value distinct from frontend, and all resources with no labels with the tier key. One could filter for resources in production excluding frontend using the comma operator: environment=production,tier!=frontendOne usage scenario for equality-based label requirement is for Pods to specify node selection criteria. For example, the sample Pod below selects nodes where the accelerator label exists and is set to nvidia-tesla-p100.apiVersion: v1 kind: Pod metadata: name: cuda-test spec: containers: - name: cuda-test image: "registry.k8s.io/cuda-vector-add:v0.1" resources: limits: nvidia.com/gpu: 1 nodeSelector: accelerator: nvidia-tesla-p100 Set-based requirementSet-based label requirements allow filtering keys according to a set of values. Three kinds of operators are supported: in,notin and exists (only the key identifier). For example:environment in (production, qa) tier notin (frontend, backend) partition !partition The first example selects all resources with key equal to environment and value equal to production or qa.The second example selects all resources with key equal to tier and values other than frontend and backend, and all resources with no labels with the tier key.The third example selects all resources including a label with key partition; no values are checked.The fourth example selects all resources without a label with key partition; no values are checked.Similarly the comma separator acts as an AND operator. So filtering resources with a partition key (no matter the value) and with environment different than qa can be achieved using partition,environment notin (qa). The set-based label selector is a general form of equality since environment=production is equivalent to environment in (production); similarly for != and notin.Set-based requirements can be mixed with equality-based requirements. For example: partition in (customerA, customerB),environment!=qa.APILIST and WATCH filteringFor list and watch operations, you can specify label selectors to filter the sets of objects returned; you specify the filter using a query parameter. (To learn in detail about watches in Kubernetes, read efficient detection of changes). Both requirements are permitted (presented here as they would appear in a URL query string):equality-based requirements: ?labelSelector=environment%3Dproduction,tier%3Dfrontendset-based requirements: ?labelSelector=environment+in+%28production%2Cqa%29%2Ctier+in+%28frontend%29Both label selector styles can be used to list or watch resources via a REST client. For example, targeting apiserver with kubectl and using equality-based one may write:kubectl get pods -l environment=production,tier=frontend or using set-based requirements:kubectl get pods -l 'environment in (production),tier in (frontend)' As already mentioned set-based requirements are more expressive. For instance, they can implement the OR operator on values:kubectl get pods -l 'environment in (production, qa)' or restricting negative matching via notin operator:kubectl get pods -l 'environment,environment notin (frontend)' Set references in API objectsSome Kubernetes objects, such as services and replicationcontrollers, also use label selectors to specify sets of other resources, such as pods.Service and ReplicationControllerThe set of pods that a service targets is defined with a label selector. Similarly, the population of pods that a replicationcontroller should manage is also defined with a label selector.Label selectors for both objects are defined in json or yaml files using maps, and only equality-based requirement selectors are supported:"selector": { "component" : "redis", } orselector: component: redis This selector (respectively in json or yaml format) is equivalent to component=redis or component in (redis).Resources that support set-based requirementsNewer resources, such as Job, Deployment, ReplicaSet, and DaemonSet, support set-based requirements as well.selector: matchLabels: component: redis matchExpressions: - { key: tier, operator: In, values: [cache] } - { key: environment, operator: NotIn, values: [dev] } matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels map is equivalent to an element of matchExpressions, whose key field is "key", the operator is "In", and the values array contains only "value". matchExpressions is a list of pod selector requirements. Valid operators include In, NotIn, Exists, and DoesNotExist. The values set must be non-empty in the case of In and NotIn. All of the requirements, from both matchLabels and matchExpressions are ANDed together -- they must all be satisfied in order to match.Selecting sets of nodesOne use case for selecting over labels is to constrain the set of nodes onto which a pod can schedule. See the documentation on node selection for more information.Using labels effectivelyYou can apply a single label to any resources, but this is not always the best practice. There are many scenarios where multiple labels should be used to distinguish resource sets from one another.For instance, different applications would use different values for the app label, but a multi-tier application, such as the guestbook example, would additionally need to distinguish each tier. The frontend could carry the following labels:labels: app: guestbook tier: frontend while the Redis master and replica would have different tier labels, and perhaps even an additional role label:labels: app: guestbook tier: backend role: master andlabels: app: guestbook tier: backend role: replica The labels allow for slicing and dicing the resources along any dimension specified by a label:kubectl apply -f examples/guestbook/all-in-one/guestbook-all-in-one.yaml kubectl get pods -Lapp -Ltier -Lrole NAME READY STATUS RESTARTS AGE APP TIER ROLE guestbook-fe-4nlpb 1/1 Running 0 1m guestbook frontend <none> guestbook-fe-ght6d 1/1 Running 0 1m guestbook frontend <none> guestbook-fe-jpy62 1/1 Running 0 1m guestbook frontend <none> guestbook-redis-master-5pg3b 1/1 Running 0 1m guestbook backend master guestbook-redis-replica-2q2yf 1/1 Running 0 1m guestbook backend replica guestbook-redis-replica-qgazl 1/1 Running 0 1m guestbook backend replica my-nginx-divi2 1/1 Running 0 29m nginx <none> <none> my-nginx-o0ef1 1/1 Running 0 29m nginx <none> <none> kubectl get pods -lapp=guestbook,role=replica NAME READY STATUS RESTARTS AGE guestbook-redis-replica-2q2yf 1/1 Running 0 3m guestbook-redis-replica-qgazl 1/1 Running 0 3m Updating labelsSometimes you may want to relabel existing pods and other resources before creating new resources. This can be done with kubectl label. For example, if you want to label all your NGINX Pods as frontend tier, run:kubectl label pods -l app=nginx tier=fe pod/my-nginx-2035384211-j5fhi labeled pod/my-nginx-2035384211-u2c7e labeled pod/my-nginx-2035384211-u3t6x labeled This first filters all pods with the label "app=nginx", and then labels them with the "tier=fe". To see the pods you labeled, run:kubectl get pods -l app=nginx -L tier NAME READY STATUS RESTARTS AGE TIER my-nginx-2035384211-j5fhi 1/1 Running 0 23m fe my-nginx-2035384211-u2c7e 1/1 Running 0 23m fe my-nginx-2035384211-u3t6x 1/1 Running 0 23m fe This outputs all "app=nginx" pods, with an additional label column of pods' tier (specified with -L or --label-columns).For more information, please see kubectl label.What's nextLearn how to add a label to a nodeFind Well-known labels, Annotations and TaintsSee Recommended labelsEnforce Pod Security Standards with Namespace LabelsRead a blog on Writing a Controller for Pod Labels


# LabelSelector

LabelSelectorA label selector is a label query over a set of resources.import "k8s.io/apimachinery/pkg/apis/meta/v1"A label selector is a label query over a set of resources. The result of matchLabels and matchExpressions are ANDed. An empty label selector matches all objects. A null label selector matches no objects.matchExpressions ([]LabelSelectorRequirement)Atomic: will be replaced during a mergematchExpressions is a list of label selector requirements. The requirements are ANDed.A label selector requirement is a selector that contains values, a key, and an operator that relates the key and values.matchExpressions.key (string), requiredkey is the label key that the selector applies to.matchExpressions.operator (string), requiredoperator represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists and DoesNotExist.matchExpressions.values ([]string)Atomic: will be replaced during a mergevalues is an array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. This array is replaced during a strategic merge patch.matchLabels (map[string]string)matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels map is equivalent to an element of matchExpressions, whose key field is "key", the operator is "In", and the values array contains only "value". The requirements are ANDed.This page is automatically generated.If you plan to report an issue with this page, mention that the page is auto-generated in your issue description. The fix may need to happen elsewhere in the Kubernetes project.


# Viewing Site Analytics

Viewing Site AnalyticsThis page contains information about the kubernetes.io analytics dashboard.View the dashboard.This dashboard is built using Google Looker Studio and shows information collected on kubernetes.io using Google Analytics 4 since August 2022.Using the dashboardBy default, the dashboard shows all collected analytics for the past 30 days. Use the date selector to see data from a different date range. Other filtering options allow you to view data based on user location, the device used to access the site, the translation of the docs used, and more.If you notice an issue with this dashboard, or would like to request any improvements, please open an issue.


# Configure Quotas for API Objects

Configure Quotas for API ObjectsThis page shows how to configure quotas for API objects, including PersistentVolumeClaims and Services. A quota restricts the number of objects, of a particular type, that can be created in a namespace. You specify quotas in a ResourceQuota object.Before you beginYou need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. It is recommended to run this tutorial on a cluster with at least two nodes that are not acting as control plane hosts. If you do not already have a cluster, you can create one by using minikube or you can use one of these Kubernetes playgrounds:iximiuz LabsKillercodaKodeKloudPlay with KubernetesTo check the version, enter kubectl version.Create a namespaceCreate a namespace so that the resources you create in this exercise are isolated from the rest of your cluster.kubectl create namespace quota-object-example Create a ResourceQuotaHere is the configuration file for a ResourceQuota object:admin/resource/quota-objects.yaml apiVersion: v1 kind: ResourceQuota metadata: name: object-quota-demo spec: hard: persistentvolumeclaims: "1" services.loadbalancers: "2" services.nodeports: "0" Create the ResourceQuota:kubectl apply -f https://k8s.io/examples/admin/resource/quota-objects.yaml --namespace=quota-object-example View detailed information about the ResourceQuota:kubectl get resourcequota object-quota-demo --namespace=quota-object-example --output=yaml The output shows that in the quota-object-example namespace, there can be at most one PersistentVolumeClaim, at most two Services of type LoadBalancer, and no Services of type NodePort.status: hard: persistentvolumeclaims: "1" services.loadbalancers: "2" services.nodeports: "0" used: persistentvolumeclaims: "0" services.loadbalancers: "0" services.nodeports: "0" Create a PersistentVolumeClaimHere is the configuration file for a PersistentVolumeClaim object:admin/resource/quota-objects-pvc.yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: pvc-quota-demo spec: storageClassName: manual accessModes: - ReadWriteOnce resources: requests: storage: 3Gi Create the PersistentVolumeClaim:kubectl apply -f https://k8s.io/examples/admin/resource/quota-objects-pvc.yaml --namespace=quota-object-example Verify that the PersistentVolumeClaim was created:kubectl get persistentvolumeclaims --namespace=quota-object-example The output shows that the PersistentVolumeClaim exists and has status Pending:NAME STATUS pvc-quota-demo Pending Attempt to create a second PersistentVolumeClaimHere is the configuration file for a second PersistentVolumeClaim:admin/resource/quota-objects-pvc-2.yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: pvc-quota-demo-2 spec: storageClassName: manual accessModes: - ReadWriteOnce resources: requests: storage: 4Gi Attempt to create the second PersistentVolumeClaim:kubectl apply -f https://k8s.io/examples/admin/resource/quota-objects-pvc-2.yaml --namespace=quota-object-example The output shows that the second PersistentVolumeClaim was not created, because it would have exceeded the quota for the namespace.persistentvolumeclaims "pvc-quota-demo-2" is forbidden: exceeded quota: object-quota-demo, requested: persistentvolumeclaims=1, used: persistentvolumeclaims=1, limited: persistentvolumeclaims=1 NotesThese are the strings used to identify API resources that can be constrained by quotas:StringAPI Object"pods"Pod"services"Service"replicationcontrollers"ReplicationController"resourcequotas"ResourceQuota"secrets"Secret"configmaps"ConfigMap"persistentvolumeclaims"PersistentVolumeClaim"services.nodeports"Service of type NodePort"services.loadbalancers"Service of type LoadBalancerClean upDelete your namespace:kubectl delete namespace quota-object-example What's nextFor cluster administratorsConfigure Default Memory Requests and Limits for a NamespaceConfigure Default CPU Requests and Limits for a NamespaceConfigure Minimum and Maximum Memory Constraints for a NamespaceConfigure Minimum and Maximum CPU Constraints for a NamespaceConfigure Memory and CPU Quotas for a NamespaceConfigure a Pod Quota for a NamespaceFor app developersAssign Memory Resources to Containers and PodsAssign CPU Resources to Containers and PodsAssign Pod-level CPU and memory resourcesConfigure Quality of Service for Pods


# Debug Services

Debug ServicesAn issue that comes up rather frequently for new installations of Kubernetes is that a Service is not working properly. You've run your Pods through a Deployment (or other workload controller) and created a Service, but you get no response when you try to access it. This document will hopefully help you to figure out what's going wrong.Running commands in a PodFor many steps here you will want to see what a Pod running in the cluster sees. The simplest way to do this is to run an interactive busybox Pod:kubectl run -it --rm --restart=Never busybox --image=gcr.io/google-containers/busybox sh Note:If you don't see a command prompt, try pressing enter.If you already have a running Pod that you prefer to use, you can run a command in it using:kubectl exec <POD-NAME> -c <CONTAINER-NAME> -- <COMMAND> SetupFor the purposes of this walk-through, let's run some Pods. Since you're probably debugging your own Service you can substitute your own details, or you can follow along and get a second data point.kubectl create deployment hostnames --image=registry.k8s.io/serve_hostname deployment.apps/hostnames created kubectl commands will print the type and name of the resource created or mutated, which can then be used in subsequent commands.Let's scale the deployment to 3 replicas.kubectl scale deployment hostnames --replicas=3 deployment.apps/hostnames scaled Note that this is the same as if you had started the Deployment with the following YAML:apiVersion: apps/v1 kind: Deployment metadata: labels: app: hostnames name: hostnames spec: selector: matchLabels: app: hostnames replicas: 3 template: metadata: labels: app: hostnames spec: containers: - name: hostnames image: registry.k8s.io/serve_hostname The label "app" is automatically set by kubectl create deployment to the name of the Deployment.You can confirm your Pods are running:kubectl get pods -l app=hostnames NAME READY STATUS RESTARTS AGE hostnames-632524106-bbpiw 1/1 Running 0 2m hostnames-632524106-ly40y 1/1 Running 0 2m hostnames-632524106-tlaok 1/1 Running 0 2m You can also confirm that your Pods are serving. You can get the list of Pod IP addresses and test them directly.kubectl get pods -l app=hostnames \ -o go-template='{{range .items}}{{.status.podIP}}{{"\n"}}{{end}}' 10.244.0.5 10.244.0.6 10.244.0.7 The example container used for this walk-through serves its own hostname via HTTP on port 9376, but if you are debugging your own app, you'll want to use whatever port number your Pods are listening on.From within a pod:for ep in 10.244.0.5:9376 10.244.0.6:9376 10.244.0.7:9376; do wget -qO- $ep done This should produce something like:hostnames-632524106-bbpiw hostnames-632524106-ly40y hostnames-632524106-tlaok If you are not getting the responses you expect at this point, your Pods might not be healthy or might not be listening on the port you think they are. You might find kubectl logs to be useful for seeing what is happening, or perhaps you need to kubectl exec directly into your Pods and debug from there.Assuming everything has gone to plan so far, you can start to investigate why your Service doesn't work.Does the Service exist?The astute reader will have noticed that you did not actually create a Service yet - that is intentional. This is a step that sometimes gets forgotten, and is the first thing to check.What would happen if you tried to access a non-existent Service? If you have another Pod that consumes this Service by name you would get something like:wget -O- hostnames Resolving hostnames (hostnames)... failed: Name or service not known. wget: unable to resolve host address 'hostnames' The first thing to check is whether that Service actually exists:kubectl get svc hostnames No resources found. Error from server (NotFound): services "hostnames" not found Let's create the Service. As before, this is for the walk-through - you can use your own Service's details here.kubectl expose deployment hostnames --port=80 --target-port=9376 service/hostnames exposed And read it back:kubectl get svc hostnames NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE hostnames ClusterIP 10.0.1.175 <none> 80/TCP 5s Now you know that the Service exists.As before, this is the same as if you had started the Service with YAML:apiVersion: v1 kind: Service metadata: labels: app: hostnames name: hostnames spec: selector: app: hostnames ports: - name: default protocol: TCP port: 80 targetPort: 9376 In order to highlight the full range of configuration, the Service you created here uses a different port number than the Pods. For many real-world Services, these values might be the same.Any Network Policy Ingress rules affecting the target Pods?If you have deployed any Network Policy Ingress rules which may affect incoming traffic to hostnames-* Pods, these need to be reviewed.Please refer to Network Policies for more details.Does the Service work by DNS name?One of the most common ways that clients consume a Service is through a DNS name.From a Pod in the same Namespace:nslookup hostnames Address 1: 10.0.0.10 kube-dns.kube-system.svc.cluster.local Name: hostnames Address 1: 10.0.1.175 hostnames.default.svc.cluster.local If this fails, perhaps your Pod and Service are in different Namespaces, try a namespace-qualified name (again, from within a Pod):nslookup hostnames.default Address 1: 10.0.0.10 kube-dns.kube-system.svc.cluster.local Name: hostnames.default Address 1: 10.0.1.175 hostnames.default.svc.cluster.local If this works, you'll need to adjust your app to use a cross-namespace name, or run your app and Service in the same Namespace. If this still fails, try a fully-qualified name:nslookup hostnames.default.svc.cluster.local Address 1: 10.0.0.10 kube-dns.kube-system.svc.cluster.local Name: hostnames.default.svc.cluster.local Address 1: 10.0.1.175 hostnames.default.svc.cluster.local Note the suffix here: "default.svc.cluster.local". The "default" is the Namespace you're operating in. The "svc" denotes that this is a Service. The "cluster.local" is your cluster domain, which COULD be different in your own cluster.You can also try this from a Node in the cluster:Note:10.0.0.10 is the cluster's DNS Service IP, yours might be different.nslookup hostnames.default.svc.cluster.local 10.0.0.10 Server: 10.0.0.10 Address: 10.0.0.10#53 Name: hostnames.default.svc.cluster.local Address: 10.0.1.175 If you are able to do a fully-qualified name lookup but not a relative one, you need to check that your /etc/resolv.conf file in your Pod is correct. From within a Pod:cat /etc/resolv.conf You should see something like:nameserver 10.0.0.10 search default.svc.cluster.local svc.cluster.local cluster.local example.com options ndots:5 The nameserver line must indicate your cluster's DNS Service. This is passed into kubelet with the --cluster-dns flag.The search line must include an appropriate suffix for you to find the Service name. In this case it is looking for Services in the local Namespace ("default.svc.cluster.local"), Services in all Namespaces ("svc.cluster.local"), and lastly for names in the cluster ("cluster.local"). Depending on your own install you might have additional records after that (up to 6 total). The cluster suffix is passed into kubelet with the --cluster-domain flag. Throughout this document, the cluster suffix is assumed to be "cluster.local". Your own clusters might be configured differently, in which case you should change that in all of the previous commands.The options line must set ndots high enough that your DNS client library considers search paths at all. Kubernetes sets this to 5 by default, which is high enough to cover all of the DNS names it generates.Does any Service work by DNS name?If the above still fails, DNS lookups are not working for your Service. You can take a step back and see what else is not working. The Kubernetes master Service should always work. From within a Pod:nslookup kubernetes.default Server: 10.0.0.10 Address 1: 10.0.0.10 kube-dns.kube-system.svc.cluster.local Name: kubernetes.default Address 1: 10.0.0.1 kubernetes.default.svc.cluster.local If this fails, please see the kube-proxy section of this document, or even go back to the top of this document and start over, but instead of debugging your own Service, debug the DNS Service.Does the Service work by IP?Assuming you have confirmed that DNS works, the


# Configuring a cgroup driver

Configuring a cgroup driverThis page explains how to configure the kubelet's cgroup driver to match the container runtime cgroup driver for kubeadm clusters.Before you beginYou should be familiar with the Kubernetes container runtime requirements.Configuring the container runtime cgroup driverThe Container runtimes page explains that the systemd driver is recommended for kubeadm based setups instead of the kubelet's default cgroupfs driver, because kubeadm manages the kubelet as a systemd service.The page also provides details on how to set up a number of different container runtimes with the systemd driver by default.Configuring the kubelet cgroup driverkubeadm allows you to pass a KubeletConfiguration structure during kubeadm init. This KubeletConfiguration can include the cgroupDriver field which controls the cgroup driver of the kubelet.Note:In v1.22 and later, if the user does not set the cgroupDriver field under KubeletConfiguration, kubeadm defaults it to systemd.In Kubernetes v1.28, you can enable automatic detection of the cgroup driver as an alpha feature. See systemd cgroup driver for more details.A minimal example of configuring the field explicitly:# kubeadm-config.yaml kind: ClusterConfiguration apiVersion: kubeadm.k8s.io/v1beta4 kubernetesVersion: v1.21.0 --- kind: KubeletConfiguration apiVersion: kubelet.config.k8s.io/v1beta1 cgroupDriver: systemd Such a configuration file can then be passed to the kubeadm command:kubeadm init --config kubeadm-config.yaml Note:Kubeadm uses the same KubeletConfiguration for all nodes in the cluster. The KubeletConfiguration is stored in a ConfigMap object under the kube-system namespace.Executing the sub commands init, join and upgrade would result in kubeadm writing the KubeletConfiguration as a file under /var/lib/kubelet/config.yaml and passing it to the local node kubelet.On each node, kubeadm detects the CRI socket and stores its details into the /var/lib/kubelet/instance-config.yaml file. When executing the init, join, or upgrade subcommands, kubeadm patches the containerRuntimeEndpoint value from this instance configuration into /var/lib/kubelet/config.yaml.Using the cgroupfs driverTo use cgroupfs and to prevent kubeadm upgrade from modifying the KubeletConfiguration cgroup driver on existing setups, you must be explicit about its value. This applies to a case where you do not wish future versions of kubeadm to apply the systemd driver by default.See the below section on "Modify the kubelet ConfigMap" for details on how to be explicit about the value.If you wish to configure a container runtime to use the cgroupfs driver, you must refer to the documentation of the container runtime of your choice.Migrating to the systemd driverTo change the cgroup driver of an existing kubeadm cluster from cgroupfs to systemd in-place, a similar procedure to a kubelet upgrade is required. This must include both steps outlined below.Note:Alternatively, it is possible to replace the old nodes in the cluster with new ones that use the systemd driver. This requires executing only the first step below before joining the new nodes and ensuring the workloads can safely move to the new nodes before deleting the old nodes.Modify the kubelet ConfigMapCall kubectl edit cm kubelet-config -n kube-system.Either modify the existing cgroupDriver value or add a new field that looks like this:cgroupDriver: systemd This field must be present under the kubelet: section of the ConfigMap.Update the cgroup driver on all nodesFor each node in the cluster:Drain the node using kubectl drain <node-name> --ignore-daemonsetsStop the kubelet using systemctl stop kubeletStop the container runtimeModify the container runtime cgroup driver to systemdSet cgroupDriver: systemd in /var/lib/kubelet/config.yamlStart the container runtimeStart the kubelet using systemctl start kubeletUncordon the node using kubectl uncordon <node-name>Execute these steps on nodes one at a time to ensure workloads have sufficient time to schedule on different nodes.Once the process is complete ensure that all nodes and workloads are healthy.


# Network Plugins

Network PluginsKubernetes (version 1.3 through to the latest 1.34, and likely onwards) lets you use Container Network Interface (CNI) plugins for cluster networking. You must use a CNI plugin that is compatible with your cluster and that suits your needs. Different plugins are available (both open- and closed- source) in the wider Kubernetes ecosystem.A CNI plugin is required to implement the Kubernetes network model.You must use a CNI plugin that is compatible with the v0.4.0 or later releases of the CNI specification. The Kubernetes project recommends using a plugin that is compatible with the v1.0.0 CNI specification (plugins can be compatible with multiple spec versions).InstallationA Container Runtime, in the networking context, is a daemon on a node configured to provide CRI Services for kubelet. In particular, the Container Runtime must be configured to load the CNI plugins required to implement the Kubernetes network model.Note:Prior to Kubernetes 1.24, the CNI plugins could also be managed by the kubelet using the cni-bin-dir and network-plugin command-line parameters. These command-line parameters were removed in Kubernetes 1.24, with management of the CNI no longer in scope for kubelet.See Troubleshooting CNI plugin-related errors if you are facing issues following the removal of dockershim.For specific information about how a Container Runtime manages the CNI plugins, see the documentation for that Container Runtime, for example:containerdCRI-OFor specific information about how to install and manage a CNI plugin, see the documentation for that plugin or networking provider.Network Plugin RequirementsLoopback CNIIn addition to the CNI plugin installed on the nodes for implementing the Kubernetes network model, Kubernetes also requires the container runtimes to provide a loopback interface lo, which is used for each sandbox (pod sandboxes, vm sandboxes, ...). Implementing the loopback interface can be accomplished by re-using the CNI loopback plugin. or by developing your own code to achieve this (see this example from CRI-O).Support hostPortThe CNI networking plugin supports hostPort. You can use the official portmap plugin offered by the CNI plugin team or use your own plugin with portMapping functionality.If you want to enable hostPort support, you must specify portMappings capability in your cni-conf-dir. For example:{ "name": "k8s-pod-network", "cniVersion": "0.4.0", "plugins": [ { "type": "calico", "log_level": "info", "datastore_type": "kubernetes", "nodename": "127.0.0.1", "ipam": { "type": "host-local", "subnet": "usePodCidr" }, "policy": { "type": "k8s" }, "kubernetes": { "kubeconfig": "/etc/cni/net.d/calico-kubeconfig" } }, { "type": "portmap", "capabilities": {"portMappings": true}, "externalSetMarkChain": "KUBE-MARK-MASQ" } ] } Support traffic shapingExperimental FeatureThe CNI networking plugin also supports pod ingress and egress traffic shaping. You can use the official bandwidth plugin offered by the CNI plugin team or use your own plugin with bandwidth control functionality.If you want to enable traffic shaping support, you must add the bandwidth plugin to your CNI configuration file (default /etc/cni/net.d) and ensure that the binary is included in your CNI bin dir (default /opt/cni/bin).{ "name": "k8s-pod-network", "cniVersion": "0.4.0", "plugins": [ { "type": "calico", "log_level": "info", "datastore_type": "kubernetes", "nodename": "127.0.0.1", "ipam": { "type": "host-local", "subnet": "usePodCidr" }, "policy": { "type": "k8s" }, "kubernetes": { "kubeconfig": "/etc/cni/net.d/calico-kubeconfig" } }, { "type": "bandwidth", "capabilities": {"bandwidth": true} } ] } Now you can add the kubernetes.io/ingress-bandwidth and kubernetes.io/egress-bandwidth annotations to your Pod. For example:apiVersion: v1 kind: Pod metadata: annotations: kubernetes.io/ingress-bandwidth: 1M kubernetes.io/egress-bandwidth: 1M ... What's nextLearn more about Cluster NetworkingLearn more about Network PoliciesLearn about the Troubleshooting CNI plugin-related errors


# Define Environment Variable Values Using An Init Container

Define Environment Variable Values Using An Init ContainerFEATURE STATE: Kubernetes v1.34 [alpha] (enabled by default: false)This page show how to configure environment variables for containers in a Pod via file.Before you beginYou need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. It is recommended to run this tutorial on a cluster with at least two nodes that are not acting as control plane hosts. If you do not already have a cluster, you can create one by using minikube or you can use one of these Kubernetes playgrounds:iximiuz LabsKillercodaKodeKloudPlay with KubernetesYour Kubernetes server must be version v1.34.To check the version, enter kubectl version.How the design worksIn this exercise, you will create a Pod that sources environment variables from files, projecting these values into the running container.pods/inject/envars-file-container.yaml apiVersion: v1 kind: Pod metadata: name: envfile-test-pod spec: initContainers: - name: setup-envfile image: nginx command: ['sh', '-c', 'echo "DB_ADDRESS=address\nREST_ENDPOINT=endpoint" > /data/config.env'] volumeMounts: - name: config mountPath: /data containers: - name: use-envfile image: nginx command: [ "/bin/sh", "-c", "env" ] env: - name: DB_ADDRESS valueFrom: fileKeyRef: path: config.env volumeName: config key: DB_ADDRESS optional: false restartPolicy: Never volumes: - name: config emptyDir: {}In this manifest, you can see the initContainer mounts an emptyDir volume and writes environment variables to a file within it, and the regular containers reference both the file and the environment variable key through the fileKeyRef field without needing to mount the volume. When optional field is set to false, the specified key in fileKeyRef must exist in the environment variables file.The volume will only be mounted to the container that writes to the file (initContainer), while the consumer container that consumes the environment variable will not have the volume mounted.The env file format adheres to the kubernetes env file standard.During container initialization, the kubelet retrieves environment variables from specified files in the emptyDir volume and exposes them to the container.Note:All container types (initContainers, regular containers, sidecars containers, and ephemeral containers) support environment variable loading from files.While these environment variables can store sensitive information, emptyDir volumes don't provide the same protection mechanisms as dedicated Secret objects. Therefore, exposing confidential environment variables to containers through this feature is not considered a security best practice.Create the Pod:kubectl apply -f https://k8s.io/examples/pods/inject/envars-file-container.yaml Verify that the container in the Pod is running:# If the new Pod isn't yet healthy, rerun this command a few times. kubectl get pods Check container logs for environment variables:kubectl logs dapi-test-pod -c use-envfile | grep DB_ADDRESS The output shows the values of selected environment variables:DB_ADDRESS=address Env File SyntaxThe format of Kubernetes env files originates from .env files.In a shell environment, .env files are typically loaded using the source .env command.For Kubernetes, the defined env file format adheres to stricter syntax rules:Blank Lines: Blank lines are ignored.Leading Spaces: Leading spaces on all lines are ignored.Variable Declaration: Variables must be declared as VAR=VAL. Spaces surrounding = and trailing spaces are ignored.VAR=VAL → VAL Comments: Lines beginning with # are treated as comments and ignored.# comment VAR=VAL → VAL VAR=VAL # not a comment → VAL # not a comment Line Continuation: A backslash (\) at the end of a variable declaration line indicates the value continues on the next line. The lines are joined with a single space.VAR=VAL \ VAL2 → VAL VAL2 What's nextLearn more about environment variables.Read Defining Environment Variables for a ContainerRead Expose Pod Information to Containers Through Environment Variables


# Control Topology Management Policies on a node

Control Topology Management Policies on a nodeFEATURE STATE: Kubernetes v1.27 [stable]An increasing number of systems leverage a combination of CPUs and hardware accelerators to support latency-critical execution and high-throughput parallel computation. These include workloads in fields such as telecommunications, scientific computing, machine learning, financial services and data analytics. Such hybrid systems comprise a high performance environment.In order to extract the best performance, optimizations related to CPU isolation, memory and device locality are required. However, in Kubernetes, these optimizations are handled by a disjoint set of components.Topology Manager is a kubelet component that aims to coordinate the set of components that are responsible for these optimizations.Before you beginYou need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. It is recommended to run this tutorial on a cluster with at least two nodes that are not acting as control plane hosts. If you do not already have a cluster, you can create one by using minikube or you can use one of these Kubernetes playgrounds:iximiuz LabsKillercodaKodeKloudPlay with KubernetesYour Kubernetes server must be at or later than version v1.18.To check the version, enter kubectl version.How topology manager worksPrior to the introduction of Topology Manager, the CPU and Device Manager in Kubernetes make resource allocation decisions independently of each other. This can result in undesirable allocations on multiple-socketed systems, and performance/latency sensitive applications will suffer due to these undesirable allocations. Undesirable in this case meaning, for example, CPUs and devices being allocated from different NUMA Nodes, thus incurring additional latency.The Topology Manager is a kubelet component, which acts as a source of truth so that other kubelet components can make topology aligned resource allocation choices.The Topology Manager provides an interface for components, called Hint Providers, to send and receive topology information. The Topology Manager has a set of node level policies which are explained below.The Topology Manager receives topology information from the Hint Providers as a bitmask denoting NUMA Nodes available and a preferred allocation indication. The Topology Manager policies perform a set of operations on the hints provided and converge on the hint determined by the policy to give the optimal result. If an undesirable hint is stored, the preferred field for the hint will be set to false. In the current policies preferred is the narrowest preferred mask. The selected hint is stored as part of the Topology Manager. Depending on the policy configured, the pod can be accepted or rejected from the node based on the selected hint. The hint is then stored in the Topology Manager for use by the Hint Providers when making the resource allocation decisions.Windows SupportFEATURE STATE: Kubernetes v1.32 [alpha] (enabled by default: false)The Topology Manager support can be enabled on Windows by using the WindowsCPUAndMemoryAffinity feature gate and it requires support in the container runtime.Topology manager scopes and policiesThe Topology Manager currently:aligns Pods of all QoS classes.aligns the requested resources that Hint Provider provides topology hints for.If these conditions are met, the Topology Manager will align the requested resources.In order to customize how this alignment is carried out, the Topology Manager provides two distinct options: scope and policy.The scope defines the granularity at which you would like resource alignment to be performed, for example, at the pod or container level. And the policy defines the actual policy used to carry out the alignment, for example, best-effort, restricted, and single-numa-node. Details on the various scopes and policies available today can be found below.Note:To align CPU resources with other requested resources in a Pod spec, the CPU Manager should be enabled and proper CPU Manager policy should be configured on a Node. See Control CPU Management Policies on the Node.Note:To align memory (and hugepages) resources with other requested resources in a Pod spec, the Memory Manager should be enabled and proper Memory Manager policy should be configured on a Node. Refer to Memory Manager documentation.Topology manager scopesThe Topology Manager can deal with the alignment of resources in a couple of distinct scopes:container (default)podEither option can be selected at a time of the kubelet startup, by setting the topologyManagerScope in the kubelet configuration file.container scopeThe container scope is used by default. You can also explicitly set the topologyManagerScope to container in the kubelet configuration file.Within this scope, the Topology Manager performs a number of sequential resource alignments, i.e., for each container (in a pod) a separate alignment is computed. In other words, there is no notion of grouping the containers to a specific set of NUMA nodes, for this particular scope. In effect, the Topology Manager performs an arbitrary alignment of individual containers to NUMA nodes.The notion of grouping the containers was endorsed and implemented on purpose in the following scope, for example the pod scope.pod scopeTo select the pod scope, set topologyManagerScope in the kubelet configuration file to pod.This scope allows for grouping all containers in a pod to a common set of NUMA nodes. That is, the Topology Manager treats a pod as a whole and attempts to allocate the entire pod (all containers) to either a single NUMA node or a common set of NUMA nodes. The following examples illustrate the alignments produced by the Topology Manager on different occasions:all containers can be and are allocated to a single NUMA node;all containers can be and are allocated to a shared set of NUMA nodes.The total amount of particular resource demanded for the entire pod is calculated according to effective requests/limits formula, and thus, this total value is equal to the maximum of:the sum of all app container requests,the maximum of init container requests,for a resource.Using the pod scope in tandem with single-numa-node Topology Manager policy is specifically valuable for workloads that are latency sensitive or for high-throughput applications that perform IPC. By combining both options, you are able to place all containers in a pod onto a single NUMA node; hence, the inter-NUMA communication overhead can be eliminated for that pod.In the case of single-numa-node policy, a pod is accepted only if a suitable set of NUMA nodes is present among possible allocations. Reconsider the example above:a set containing only a single NUMA node - it leads to pod being admitted,whereas a set containing more NUMA nodes - it results in pod rejection (because instead of one NUMA node, two or more NUMA nodes are required to satisfy the allocation).To recap, the Topology Manager first computes a set of NUMA nodes and then tests it against the Topology Manager policy, which either leads to the rejection or admission of the pod.Topology manager policiesThe Topology Manager supports four allocation policies. You can set a policy via a kubelet flag, --topology-manager-policy. There are four supported policies:none (default)best-effortrestrictedsingle-numa-nodeNote:If the Topology Manager is configured with the pod scope, the container, which is considered by the policy, is reflecting requirements of the entire pod, and thus each container from the pod will result with the same topology alignment decision.none policyThis is the default policy and does not perform any topology alignment.best-effort policyFor each container in a Pod, the kubelet, with best-effort topology management policy, calls each Hint Provider to discover their resource availability. Using this information, the Topology Manager stores the preferred NUMA Node affinity for that container. If the affinity is not preferred, the Topology Manager will store this and admit the pod to the node anyway.The Hint Providers can then use this information when making the resource allocation decision.restricted policyFor each container in a Pod, the kubelet, with restricted topology management policy, calls each Hint Provider to discover their resource availability. Using this information, the Topology Manager stores the preferred NUMA Node affinity for that container. If the affinity is not preferred, the Topology Manager will reject this pod from the node. This will result in a pod entering a Terminated state with a pod admission failure.Once the pod is in a Terminated state, the Kubernetes scheduler will not attempt to reschedule the pod. It is recommended to use a ReplicaSet or Deployment to trigger a redeployment of the pod. An external control loop could be also implemented to trigger a redeployment of pods that have the Topology Affinity error.If the pod is admitted, the Hint Providers can then use this information when making the resource allocation decision.single-numa-node policyFor each container in a Pod, the kubelet, with single-numa-node topology management policy, calls each Hint Provider to discover their resource availability. Using this information, the Topology Manager determines if a single NUMA Node affinity is possible. If it is, Topology Manager will store this and the Hint Providers can then use this information when making the resource allocation decision. If, however, this is not possible then the Topology Manager will reject the pod from the node. This will result in a pod in a Terminated state with a pod admission failure.Once the pod is in a Terminated state, the Kubernetes scheduler will not attempt to reschedule the pod. It is recommended to use a Deployment with replicas to trigger a redeployment of the Pod. An external control loop could be also implemented to trigger a redeployment of pods that have the Topology Affinity error.Topology manager policy optionsSupport for the Topology Manager policy options requires TopologyManagerPolicyOptions feature gate to be enabled (it is enabled by default).You can toggle groups of options on and off based upon their maturity level using the following feature gates:TopologyManagerPolicyBetaOptions default enabled. Enable to show beta-level options.TopologyManagerPolicyAlphaOptions default disabled. Enable to show alpha-level options.You will still have to enable each option using the TopologyManagerPolicyOptions kubelet option.prefer-closest-numa-nodesThe prefer-closest-numa-nodes option is GA since Kubernetes 1.32. In Kubernetes 1.34 this policy option is visible by default provided that the TopologyManagerPolicyOptions feature gate is enabled.The Topology Manager is not aware by default of NUMA distances, and does not take them into account when making Pod admission decisions. This limitation surfaces in multi-socket, as well as single-socket multi NUMA systems, and can cause significant performance degradation in latency-critical execution and high-throughput applications if the Topology Manager decides to align resources on non-adjacent NUMA nodes.If you specify the prefer-closest-numa-nodes policy option, the best-effort and restricted policies favor sets of NUMA nodes with shorter distance between them when making admission decisions.You can enable this option by adding prefer-closest-numa-nodes=true to the Topology Manager policy options.By default (without this option), the Topology Manager aligns resources on either a single NUMA node or, in the case where more than one NUMA node is required, using the minimum number of NUMA nodes.max-allowable-numa-nodes (beta)The max-allowable-numa-nodes option is beta since Kubernetes 1.31. In Kubernetes 1.34, this policy option is visible by default provided that the TopologyManagerPolicyOptions and TopologyManagerPolicyBetaOptions feature gates are enabled.The time to admit a pod is tied to the number of NUMA nodes on the physical machine. By default, Kubernetes does not run a kubelet with the Topology Manager enabled, on any (Kubernetes) node where more than 8 NUMA nodes are detected.Note:If you select the max-allowable-numa-nodes policy option, nodes with more than 8 NUMA nodes can be allowed to run with the Topology Manager enabled. The Kubernetes project only has limited data on the impact of using the Topology Manager on (Kubernetes) nodes with more than 8 NUMA nodes. Because of that lack of data, using this policy option with Kubernetes 1.34 is not recommended and is at your own risk.You can enable this option by adding max-allowable-numa-nodes=true to the Topology Manager policy options.Setting a value of max-allowable-numa-nodes does not (in and of itself) affect the latency of pod admission, but binding a Pod to a (Kubernetes) node with many NUMA does have an impact. Future, potential improvements to Kubernetes may improve Pod admission performance and the high latency that happens as the number of NUMA nodes increases.Pod interactions with topology manager policiesConsider the containers in the following Pod manifest:spec: containers: - name: nginx image: nginx This pod runs in the BestEffort QoS class because no resource requests or limits are specified.spec: containers: - name: nginx image: nginx resources: limits: memory: "200Mi" requests: memory: "100Mi" This pod runs in the Burstable QoS class because requests are less than limits.If the selected policy is anything other than none, the Topology Manager would consider these Pod specifications. The Topology Manager would consult the Hint Providers to get topology hints. In the case of the static, the CPU Manager policy would return default topology hint, because these Pods do not explicitly request CPU resources.spec: containers: - name: nginx image: nginx resources: limits: memory: "200Mi" cpu: "2" example.com/device: "1" requests: memory: "200Mi" cpu: "2" example.com/device: "1" This pod with integer CPU request runs in the Guaranteed QoS class because requests are equal to limits.spec: containers: - name: nginx image: nginx resources: limits: memory: "200Mi" cpu: "300m" example.com/device: "1" requests: memory: "200Mi" cpu: "300m" example.com/device: "1" This pod with sharing CPU request runs in the Guaranteed QoS class because requests are equal to limits.spec: containers: - name: nginx image: nginx resources: limits: example.com/deviceA: "1" example.com/deviceB: "1" requests: example.com/deviceA: "1" example.com/deviceB: "1" This pod runs in the BestEffort QoS class because there are no CPU and memory requests.The Topology Manager would consider the above pods. The Topology Manager would consult the Hint Providers, which are CPU and Device Manager to get topology hints for the pods.In the case of the Guaranteed pod with integer CPU request, the static CPU Manager policy would return topology hints relating to the exclusive CPU and the Device Manager would send back hints for the requested device.In the case of the Guaranteed pod with sharing CPU request, the static CPU Manager policy would return default topology hint as there is no exclusive CPU request and the Device Manager would send back hints for the requested device.In the above two cases of the Guaranteed pod, the none CPU Manager policy would return default topology hint.In the case of the BestEffort pod, the static CPU Manager policy would send back the default topology hint as there is no CPU request and the Device Manager would send back the hints for each of the requested devices.Using this information the Topology Manager calculates the optimal hint for the pod and stores this information, which will be used by the Hint Providers when they are making their resource assignments.Known limitationsThe maximum number of NUMA nodes that Topology Manager allows is 8. With more than 8 NUMA nodes, there will be a state explosion when trying to enumerate the possible NUMA affinities and generating their hints. See max-allowable-numa-nodes (beta) for more options.The scheduler is not topology-aware, so it is possible to be scheduled on a node and then fail on the node due to the Topology Manager.


# Good practices for Dynamic Resource Allocation as a Cluster Admin

Good practices for Dynamic Resource Allocation as a Cluster AdminThis page describes good practices when configuring a Kubernetes cluster utilizing Dynamic Resource Allocation (DRA). These instructions are for cluster administrators.Separate permissions to DRA related APIsDRA is orchestrated through a number of different APIs. Use authorization tools (like RBAC, or another solution) to control access to the right APIs depending on the persona of your user.In general, DeviceClasses and ResourceSlices should be restricted to admins and the DRA drivers. Cluster operators that will be deploying Pods with claims will need access to ResourceClaim and ResourceClaimTemplate APIs; both of these APIs are namespace scoped.DRA driver deployment and maintenanceDRA drivers are third-party applications that run on each node of your cluster to interface with the hardware of that node and Kubernetes' native DRA components. The installation procedure depends on the driver you choose, but is likely deployed as a DaemonSet to all or a selection of the nodes (using node selectors or similar mechanisms) in your cluster.Use drivers with seamless upgrade if availableDRA drivers implement the kubeletplugin package interface. Your driver may support seamless upgrades by implementing a property of this interface that allows two versions of the same DRA driver to coexist for a short time. This is only available for kubelet versions 1.33 and above and may not be supported by your driver for heterogeneous clusters with attached nodes running older versions of Kubernetes - check your driver's documentation to be sure.If seamless upgrades are available for your situation, consider using it to minimize scheduling delays when your driver updates.If you cannot use seamless upgrades, during driver downtime for upgrades you may observe that:Pods cannot start unless the claims they depend on were already prepared for use.Cleanup after the last pod which used a claim gets delayed until the driver is available again. The pod is not marked as terminated. This prevents reusing the resources used by the pod for other pods.Running pods will continue to run.Confirm your DRA driver exposes a liveness probe and utilize itYour DRA driver likely implements a gRPC socket for healthchecks as part of DRA driver good practices. The easiest way to utilize this grpc socket is to configure it as a liveness probe for the DaemonSet deploying your DRA driver. Your driver's documentation or deployment tooling may already include this, but if you are building your configuration separately or not running your DRA driver as a Kubernetes pod, be sure that your orchestration tooling restarts the DRA driver on failed healthchecks to this grpc socket. Doing so will minimize any accidental downtime of the DRA driver and give it more opportunities to self heal, reducing scheduling delays or troubleshooting time.When draining a node, drain the DRA driver as late as possibleThe DRA driver is responsible for unpreparing any devices that were allocated to Pods, and if the DRA driver is drained before Pods with claims have been deleted, it will not be able to finalize its cleanup. If you implement custom drain logic for nodes, consider checking that there are no allocated/reserved ResourceClaim or ResourceClaimTemplates before terminating the DRA driver itself.Monitor and tune components for higher load, especially in high scale environmentsControl plane component kube-scheduler and the internal ResourceClaim controller orchestrated by the component kube-controller-manager do the heavy lifting during scheduling of Pods with claims based on metadata stored in the DRA APIs. Compared to non-DRA scheduled Pods, the number of API server calls, memory, and CPU utilization needed by these components is increased for Pods using DRA claims. In addition, node local components like the DRA driver and kubelet utilize DRA APIs to allocated the hardware request at Pod sandbox creation time. Especially in high scale environments where clusters have many nodes, and/or deploy many workloads that heavily utilize DRA defined resource claims, the cluster administrator should configure the relevant components to anticipate the increased load.The effects of mistuned components can have direct or snowballing affects causing different symptoms during the Pod lifecycle. If the kube-scheduler component's QPS and burst configurations are too low, the scheduler might quickly identify a suitable node for a Pod but take longer to bind the Pod to that node. With DRA, during Pod scheduling, the QPS and Burst parameters in the client-go configuration within kube-controller-manager are critical.The specific values to tune your cluster to depend on a variety of factors like number of nodes/pods, rate of pod creation, churn, even in non-DRA environments; see the SIG Scalability README on Kubernetes scalability thresholds for more information. In scale tests performed against a DRA enabled cluster with 100 nodes, involving 720 long-lived pods (90% saturation) and 80 churn pods (10% churn, 10 times), with a job creation QPS of 10, kube-controller-manager QPS could be set to as low as 75 and Burst to 150 to meet equivalent metric targets for non-DRA deployments. At this lower bound, it was observed that the client side rate limiter was triggered enough to protect the API server from explosive burst but was high enough that pod startup SLOs were not impacted. While this is a good starting point, you can get a better idea of how to tune the different components that have the biggest effect on DRA performance for your deployment by monitoring the following metrics. For more information on all the stable metrics in Kubernetes, see the Kubernetes Metrics Reference.kube-controller-manager metricsThe following metrics look closely at the internal ResourceClaim controller managed by the kube-controller-manager component.Workqueue Add Rate: Monitor sum(rate(workqueue_adds_total{name="resource_claim"}[5m])) to gauge how quickly items are added to the ResourceClaim controller.Workqueue Depth: Track sum(workqueue_depth{endpoint="kube-controller-manager", name="resource_claim"}) to identify any backlogs in the ResourceClaim controller.Workqueue Work Duration: Observe histogram_quantile(0.99, sum(rate(workqueue_work_duration_seconds_bucket{name="resource_claim"}[5m])) by (le)) to understand the speed at which the ResourceClaim controller processes work.If you are experiencing low Workqueue Add Rate, high Workqueue Depth, and/or high Workqueue Work Duration, this suggests the controller isn't performing optimally. Consider tuning parameters like QPS, burst, and CPU/memory configurations.If you are experiencing high Workequeue Add Rate, high Workqueue Depth, but reasonable Workqueue Work Duration, this indicates the controller is processing work, but concurrency might be insufficient. Concurrency is hardcoded in the controller, so as a cluster administrator, you can tune for this by reducing the pod creation QPS, so the add rate to the resource claim workqueue is more manageable.kube-scheduler metricsThe following scheduler metrics are high level metrics aggregating performance across all Pods scheduled, not just those using DRA. It is important to note that the end-to-end metrics are ultimately influenced by the kube-controller-manager's performance in creating ResourceClaims from ResourceClainTemplates in deployments that heavily use ResourceClainTemplates.Scheduler End-to-End Duration: Monitor histogram_quantile(0.99, sum(increase(scheduler_pod_scheduling_sli_duration_seconds_bucket[5m])) by (le)).Scheduler Algorithm Latency: Track histogram_quantile(0.99, sum(increase(scheduler_scheduling_algorithm_duration_seconds_bucket[5m])) by (le)).kubelet metricsWhen a Pod bound to a node must have a ResourceClaim satisfied, kubelet calls the NodePrepareResources and NodeUnprepareResources methods of the DRA driver. You can observe this behavior from the kubelet's point of view with the following metrics.Kubelet NodePrepareResources: Monitor histogram_quantile(0.99, sum(rate(dra_operations_duration_seconds_bucket{operation_name="PrepareResources"}[5m])) by (le)).Kubelet NodeUnprepareResources: Track histogram_quantile(0.99, sum(rate(dra_operations_duration_seconds_bucket{operation_name="UnprepareResources"}[5m])) by (le)).DRA kubeletplugin operationsDRA drivers implement the kubeletplugin package interface which surfaces its own metric for the underlying gRPC operation NodePrepareResources and NodeUnprepareResources. You can observe this behavior from the point of view of the internal kubeletplugin with the following metrics.DRA kubeletplugin gRPC NodePrepareResources operation: Observe histogram_quantile(0.99, sum(rate(dra_grpc_operations_duration_seconds_bucket{method_name=~".*NodePrepareResources"}[5m])) by (le)).DRA kubeletplugin gRPC NodeUnprepareResources operation: Observe histogram_quantile(0.99, sum(rate(dra_grpc_operations_duration_seconds_bucket{method_name=~".*NodeUnprepareResources"}[5m])) by (le)).What's nextLearn more about DRARead the Kubernetes Metrics Reference


# kubectl rollout restart

kubectl rollout restartSynopsisRestart a resource. Resource rollout will be restarted. kubectl rollout restart RESOURCE Examples # Restart all deployments in the test-namespace namespace kubectl rollout restart deployment -n test-namespace # Restart a deployment kubectl rollout restart deployment/nginx # Restart a daemon set kubectl rollout restart daemonset/abc # Restart deployments with the app=nginx label kubectl rollout restart deployment --selector=app=nginx Options--allow-missing-template-keys Default: trueIf true, ignore any errors in templates when a field or map key is missing in the template. Only applies to golang and jsonpath output formats.--field-manager string Default: "kubectl-rollout"Name of the manager used to track field ownership.-f, --filename stringsFilename, directory, or URL to files identifying the resource to get from a server.-h, --helphelp for restart-k, --kustomize stringProcess the kustomization directory. This flag can't be used together with -f or -R.-o, --output stringOutput format. One of: (json, yaml, name, go-template, go-template-file, template, templatefile, jsonpath, jsonpath-as-json, jsonpath-file).-R, --recursiveProcess the directory used in -f, --filename recursively. Useful when you want to manage related manifests organized within the same directory.-l, --selector stringSelector (label query) to filter on, supports '=', '==', '!=', 'in', 'notin'.(e.g. -l key1=value1,key2=value2,key3 in (value3)). Matching objects must satisfy all of the specified label constraints.--show-managed-fieldsIf true, keep the managedFields when printing objects in JSON or YAML format.--template stringTemplate string or path to template file to use when -o=go-template, -o=go-template-file. The template format is golang templates [http://golang.org/pkg/text/template/#pkg-overview].Parent Options Inherited--as stringUsername to impersonate for the operation. User could be a regular user or a service account in a namespace.--as-group stringsGroup to impersonate for the operation, this flag can be repeated to specify multiple groups.--as-uid stringUID to impersonate for the operation.--cache-dir string Default: "$HOME/.kube/cache"Default cache directory--certificate-authority stringPath to a cert file for the certificate authority--client-certificate stringPath to a client certificate file for TLS--client-key stringPath to a client key file for TLS--cluster stringThe name of the kubeconfig cluster to use--context stringThe name of the kubeconfig context to use--disable-compressionIf true, opt-out of response compression for all requests to the server--insecure-skip-tls-verifyIf true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure--kubeconfig stringPath to the kubeconfig file to use for CLI requests.--kuberc stringPath to the kuberc file to use for preferences. This can be disabled by exporting KUBECTL_KUBERC=false feature gate or turning off the feature KUBERC=off.--match-server-versionRequire server version to match client version-n, --namespace stringIf present, the namespace scope for this CLI request--password stringPassword for basic authentication to the API server--profile string Default: "none"Name of profile to capture. One of (none|cpu|heap|goroutine|threadcreate|block|mutex)--profile-output string Default: "profile.pprof"Name of the file to write the profile to--request-timeout string Default: "0"The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests.-s, --server stringThe address and port of the Kubernetes API server--storage-driver-buffer-duration duration Default: 1m0sWrites in the storage driver will be buffered for this duration, and committed to the non memory backends as a single transaction--storage-driver-db string Default: "cadvisor"database name--storage-driver-host string Default: "localhost:8086"database host:port--storage-driver-password string Default: "root"database password--storage-driver-secureuse secure connection with database--storage-driver-table string Default: "stats"table name--storage-driver-user string Default: "root"database username--tls-server-name stringServer name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used--token stringBearer token for authentication to the API server--user stringThe name of the kubeconfig user to use--username stringUsername for basic authentication to the API server--version version[=true]--version, --version=raw prints version information and quits; --version=vX.Y.Z... sets the reported version--warnings-as-errorsTreat warnings received from the server as errors and exit with a non-zero exit codeSee Alsokubectl rollout - Manage the rollout of a resourceThis page is automatically generated.If you plan to report an issue with this page, mention that the page is auto-generated in your issue description. The fix may need to happen elsewhere in the Kubernetes project.


# Troubleshooting Clusters

Troubleshooting ClustersDebugging common cluster issues.This doc is about cluster troubleshooting; we assume you have already ruled out your application as the root cause of the problem you are experiencing. See the application troubleshooting guide for tips on application debugging. You may also visit the troubleshooting overview document for more information.For troubleshooting kubectl, refer to Troubleshooting kubectl.Listing your clusterThe first thing to debug in your cluster is if your nodes are all registered correctly.Run the following command:kubectl get nodes And verify that all of the nodes you expect to see are present and that they are all in the Ready state.To get detailed information about the overall health of your cluster, you can run:kubectl cluster-info dump Example: debugging a down/unreachable nodeSometimes when debugging it can be useful to look at the status of a node -- for example, because you've noticed strange behavior of a Pod that's running on the node, or to find out why a Pod won't schedule onto the node. As with Pods, you can use kubectl describe node and kubectl get node -o yaml to retrieve detailed information about nodes. For example, here's what you'll see if a node is down (disconnected from the network, or kubelet dies and won't restart, etc.). Notice the events that show the node is NotReady, and also notice that the pods are no longer running (they are evicted after five minutes of NotReady status).kubectl get nodes NAME STATUS ROLES AGE VERSION kube-worker-1 NotReady <none> 1h v1.23.3 kubernetes-node-bols Ready <none> 1h v1.23.3 kubernetes-node-st6x Ready <none> 1h v1.23.3 kubernetes-node-unaj Ready <none> 1h v1.23.3 kubectl describe node kube-worker-1 Name: kube-worker-1 Roles: <none> Labels: beta.kubernetes.io/arch=amd64 beta.kubernetes.io/os=linux kubernetes.io/arch=amd64 kubernetes.io/hostname=kube-worker-1 kubernetes.io/os=linux node.alpha.kubernetes.io/ttl: 0 volumes.kubernetes.io/controller-managed-attach-detach: true CreationTimestamp: Thu, 17 Feb 2022 16:46:30 -0500 Taints: node.kubernetes.io/unreachable:NoExecute node.kubernetes.io/unreachable:NoSchedule Unschedulable: false Lease: HolderIdentity: kube-worker-1 AcquireTime: <unset> RenewTime: Thu, 17 Feb 2022 17:13:09 -0500 Conditions: Type Status LastHeartbeatTime LastTransitionTime Reason Message ---- ------ ----------------- ------------------ ------ ------- NetworkUnavailable False Thu, 17 Feb 2022 17:09:13 -0500 Thu, 17 Feb 2022 17:09:13 -0500 WeaveIsUp Weave pod has set this MemoryPressure Unknown Thu, 17 Feb 2022 17:12:40 -0500 Thu, 17 Feb 2022 17:13:52 -0500 NodeStatusUnknown Kubelet stopped posting node status. DiskPressure Unknown Thu, 17 Feb 2022 17:12:40 -0500 Thu, 17 Feb 2022 17:13:52 -0500 NodeStatusUnknown Kubelet stopped posting node status. PIDPressure Unknown Thu, 17 Feb 2022 17:12:40 -0500 Thu, 17 Feb 2022 17:13:52 -0500 NodeStatusUnknown Kubelet stopped posting node status. Ready Unknown Thu, 17 Feb 2022 17:12:40 -0500 Thu, 17 Feb 2022 17:13:52 -0500 NodeStatusUnknown Kubelet stopped posting node status. Addresses: InternalIP: 192.168.0.113 Hostname: kube-worker-1 Capacity: cpu: 2 ephemeral-storage: 15372232Ki hugepages-2Mi: 0 memory: 2025188Ki pods: 110 Allocatable: cpu: 2 ephemeral-storage: 14167048988 hugepages-2Mi: 0 memory: 1922788Ki pods: 110 System Info: Machine ID: 9384e2927f544209b5d7b67474bbf92b System UUID: aa829ca9-73d7-064d-9019-df07404ad448 Boot ID: 5a295a03-aaca-4340-af20-1327fa5dab5c Kernel Version: 5.13.0-28-generic OS Image: Ubuntu 21.10 Operating System: linux Architecture: amd64 Container Runtime Version: containerd://1.5.9 Kubelet Version: v1.23.3 Kube-Proxy Version: v1.23.3 Non-terminated Pods: (4 in total) Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits Age --------- ---- ------------ ---------- --------------- ------------- --- default nginx-deployment-67d4bdd6f5-cx2nz 500m (25%) 500m (25%) 128Mi (6%) 128Mi (6%) 23m default nginx-deployment-67d4bdd6f5-w6kd7 500m (25%) 500m (25%) 128Mi (6%) 128Mi (6%) 23m kube-system kube-proxy-dnxbz 0 (0%) 0 (0%) 0 (0%) 0 (0%) 28m kube-system weave-net-gjxxp 100m (5%) 0 (0%) 200Mi (10%) 0 (0%) 28m Allocated resources: (Total limits may be over 100 percent, i.e., overcommitted.) Resource Requests Limits -------- -------- ------ cpu 1100m (55%) 1 (50%) memory 456Mi (24%) 256Mi (13%) ephemeral-storage 0 (0%) 0 (0%) hugepages-2Mi 0 (0%) 0 (0%) Events: ... kubectl get node kube-worker-1 -o yaml apiVersion: v1 kind: Node metadata: annotations: node.alpha.kubernetes.io/ttl: "0" volumes.kubernetes.io/controller-managed-attach-detach: "true" creationTimestamp: "2022-02-17T21:46:30Z" labels: beta.kubernetes.io/arch: amd64 beta.kubernetes.io/os: linux kubernetes.io/arch: amd64 kubernetes.io/hostname: kube-worker-1 kubernetes.io/os: linux name: kube-worker-1 resourceVersion: "4026" uid: 98efe7cb-2978-4a0b-842a-1a7bf12c05f8 spec: {} status: addresses: - address: 192.168.0.113 type: InternalIP - address: kube-worker-1 type: Hostname allocatable: cpu: "2" ephemeral-storage: "14167048988" hugepages-2Mi: "0" memory: 1922788Ki pods: "110" capacity: cpu: "2" ephemeral-storage: 15372232Ki hugepages-2Mi: "0" memory: 2025188Ki pods: "110" conditions: - lastHeartbeatTime: "2022-02-17T22:20:32Z" lastTransitionTime: "2022-02-17T22:20:32Z" message: Weave pod has set this reason: WeaveIsUp status: "False" type: NetworkUnavailable - lastHeartbeatTime: "2022-02-17T22:20:15Z" lastTransitionTime: "2022-02-17T22:13:25Z" message: kubelet has sufficient memory available reason: KubeletHasSufficientMemory status: "False" type: MemoryPressure - lastHeartbeatTime: "2022-02-17T22:20:15Z" lastTransitionTime: "2022-02-17T22:13:25Z" message: kubelet has no disk pressure reason: KubeletHasNoDiskPressure status: "False" type: DiskPressure - lastHeartbeatTime: "2022-02-17T22:20:15Z" lastTransitionTime: "2022-02-17T22:13:25Z" message: kubelet has sufficient PID available reason: KubeletHasSufficientPID status: "False" type: PIDPressure - lastHeartbeatTime: "2022-02-17T22:20:15Z" lastTransitionTime: "2022-02-17T22:15:15Z" message: kubelet is posting ready status reason: KubeletReady status: "True" type: Ready daemonEndpoints: kubeletEndpoint: Port: 10250 nodeInfo: architecture: amd64 bootID: 22333234-7a6b-44d4-9ce1-67e31dc7e369 containerRuntimeVersion: containerd://1.5.9 kernelVersion: 5.13.0-28-generic kubeProxyVersion: v1.23.3 kubeletVersion: v1.23.3 machineID: 9384e2927f544209b5d7b67474bbf92b operatingSystem: linux osImage: Ubuntu 21.10 systemUUID: aa829ca9-73d7-064d-9019-df07404ad448 Looking at logsFor now, digging deeper into the cluster requires logging into the relevant machines. Here are the locations of the relevant log files. On systemd-based systems, you may need to use journalctl instead of examining log files.Control Plane nodes/var/log/kube-apiserver.log - API Server, responsible for serving the API/var/log/kube-scheduler.log - Scheduler, responsible for making scheduling decisions/var/log/kube-controller-manager.log - a component that runs most Kubernetes built-in controllers, with the notable exception of scheduling (the kube-scheduler handles scheduling).Worker Nodes/var/log/kubelet.log - logs from the kubelet, responsible for running containers on the node/var/log/kube-proxy.log - logs from kube-proxy, which is responsible for directing traffic to Service endpointsCluster failure modesThis is an incomplete list of things that could go wrong, and how to adjust your cluster setup to mitigate the problems.Contributing causesVM(s) shutdownNetwork partition within cluster, or between cluster and usersCrashes in Kubernetes softwareData loss or unavailability of persistent storage (e.g. GCE PD or AWS EBS volume)Operator error, for example, misconfigured Kubernetes software or application softwareSpecific scenariosAPI server VM shutdown or apiserver crashingResultsunable to stop, update, or start new pods, services, replication controllerexisting pods and services should continue to work normally unless they depend on the Kubernetes APIAPI server backing storage lostResultsthe kube-apiserver component fails to start successfully and become healthykubelets will not be able to reach it but will continue to run the same pods and provide the same service proxyingmanual recovery or recreation of apiserver state necessary before apiserver is restartedSupporting services (node controller, replication controller manager, scheduler, etc) VM shutdown or crashescurrently those are colocated with the apiserver, and their unavailability has similar consequences as apiserverin future, these will be replicated as well and may not be co-locatedthey do not have their own persistent stateIndividual node (VM or physical machine) shuts downResultspods on that Node stop runningNetwork partitionResultspartition A thinks the nodes in partition B are down; partition B thinks the apiserver is down. (Assuming the master VM ends up in partition A.)Kubelet software faultResultscrashing kubelet cannot start new pods on the nodekubelet might delete the pods or notnode marked unhealthyreplication controllers start new pods elsewhereCluster operator errorResultsloss of pods, services, etclost of apiserver backing storeusers unable to read APIetc.MitigationsAction: Use the IaaS provider's automatic VM restarting feature for IaaS VMsMitigates: Apiserver VM shutdown or apiserver crashingMitigates: Supporting services VM shutdown or crashesAction: Use IaaS providers reliable storage (e.g. GCE PD or AWS EBS volume) for VMs with apiserver+etcdMitigates: Apiserver backing storage lostAction: Use high-availability configurationMitigates: Control plane node shutdown or control plane components (scheduler, API server, controller-manager) crashingWill tolerate one or more simultaneous node or component failuresMitigates: API server backing storage (i.e., etcd's data directory) lostAssumes HA (highly-available) etcd configurationAction: Snapshot apiserver PDs/EBS-volumes periodicallyMitigates: Apiserver backing storage lostMitigates: Some cases of operator errorMitigates: Some cases of Kubernetes software faultAction: use replication controller and services in front of podsMitigates: Node shutdownMitigates: Kubelet software faultAction: applications (containers) designed to tolerate unexpected restartsMitigates: Node shutdownMitigates: Kubelet software faultWhat's nextLearn about the metrics available in the Resource Metrics PipelineDiscover additional tools for monitoring resource usageUse Node Problem Detector to monitor node healthUse kubectl debug node to debug Kubernetes nodesUse crictl to debug Kubernetes nodesGet more information about Kubernetes auditingUse telepresence to develop and debug services locally


# Kubernetes Custom Metrics (v1beta2)

Kubernetes Custom Metrics (v1beta2)Package v1beta2 is the v1beta2 version of the custom_metrics API.Resource TypesMetricListOptionsMetricValueMetricValueListMetricListOptionsMetricListOptions is used to select metrics by their label selectorsFieldDescriptionapiVersionstringcustom.metrics.k8s.io/v1beta2kindstringMetricListOptionslabelSelectorstringA selector to restrict the list of returned objects by their labels. Defaults to everything.metricLabelSelectorstringA selector to restrict the list of returned metrics by their labelsMetricValueAppears in:MetricValueListMetricValue is the metric value for some objectFieldDescriptionapiVersionstringcustom.metrics.k8s.io/v1beta2kindstringMetricValuedescribedObject [Required]core/v1.ObjectReferencea reference to the described objectmetric [Required]MetricIdentifierNo description provided.timestamp [Required]meta/v1.Timeindicates the time at which the metrics were producedwindowSeconds [Required]int64indicates the window ([Timestamp-Window, Timestamp]) from which these metrics were calculated, when returning rate metrics calculated from cumulative metrics (or zero for non-calculated instantaneous metrics).value [Required]k8s.io/apimachinery/pkg/api/resource.Quantitythe value of the metric for thisMetricValueListMetricValueList is a list of values for a given metric for some set of objectsFieldDescriptionapiVersionstringcustom.metrics.k8s.io/v1beta2kindstringMetricValueListmetadata [Required]meta/v1.ListMetaNo description provided.items [Required][]MetricValuethe value of the metric across the described objectsMetricIdentifierAppears in:MetricValueMetricIdentifier identifies a metric by name and, optionally, selectorFieldDescriptionname [Required]stringname is the name of the given metricselectormeta/v1.LabelSelectorselector represents the label selector that could be used to select this metric, and will generally just be the selector passed in to the query used to fetch this metric. When left blank, only the metric's Name will be used to gather metrics.This page is automatically generated.If you plan to report an issue with this page, mention that the page is auto-generated in your issue description. The fix may need to happen elsewhere in the Kubernetes project.