# Pod Security Admission

Pod Security AdmissionAn overview of the Pod Security Admission Controller, which can enforce the Pod Security Standards.FEATURE STATE: Kubernetes v1.25 [stable]The Kubernetes Pod Security Standards define different isolation levels for Pods. These standards let you define how you want to restrict the behavior of pods in a clear, consistent fashion.Kubernetes offers a built-in Pod Security admission controller to enforce the Pod Security Standards. Pod security restrictions are applied at the namespace level when pods are created.Built-in Pod Security admission enforcementThis page is part of the documentation for Kubernetes v1.34. If you are running a different version of Kubernetes, consult the documentation for that release.Pod Security levelsPod Security admission places requirements on a Pod's Security Context and other related fields according to the three levels defined by the Pod Security Standards: privileged, baseline, and restricted. Refer to the Pod Security Standards page for an in-depth look at those requirements.Pod Security Admission labels for namespacesOnce the feature is enabled or the webhook is installed, you can configure namespaces to define the admission control mode you want to use for pod security in each namespace. Kubernetes defines a set of labels that you can set to define which of the predefined Pod Security Standard levels you want to use for a namespace. The label you select defines what action the control plane takes if a potential violation is detected:Pod Security Admission modesModeDescriptionenforcePolicy violations will cause the pod to be rejected.auditPolicy violations will trigger the addition of an audit annotation to the event recorded in the audit log, but are otherwise allowed.warnPolicy violations will trigger a user-facing warning, but are otherwise allowed.A namespace can configure any or all modes, or even set a different level for different modes.For each mode, there are two labels that determine the policy used:# The per-mode level label indicates which policy level to apply for the mode. # # MODE must be one of `enforce`, `audit`, or `warn`. # LEVEL must be one of `privileged`, `baseline`, or `restricted`. pod-security.kubernetes.io/<MODE>: <LEVEL> # Optional: per-mode version label that can be used to pin the policy to the # version that shipped with a given Kubernetes minor version (for example v1.34). # # MODE must be one of `enforce`, `audit`, or `warn`. # VERSION must be a valid Kubernetes minor version, or `latest`. pod-security.kubernetes.io/<MODE>-version: <VERSION> Check out Enforce Pod Security Standards with Namespace Labels to see example usage.Workload resources and Pod templatesPods are often created indirectly, by creating a workload object such as a Deployment or Job. The workload object defines a Pod template and a controller for the workload resource creates Pods based on that template. To help catch violations early, both the audit and warning modes are applied to the workload resources. However, enforce mode is not applied to workload resources, only to the resulting pod objects.ExemptionsYou can define exemptions from pod security enforcement in order to allow the creation of pods that would have otherwise been prohibited due to the policy associated with a given namespace. Exemptions can be statically configured in the Admission Controller configuration.Exemptions must be explicitly enumerated. Requests meeting exemption criteria are ignored by the Admission Controller (all enforce, audit and warn behaviors are skipped). Exemption dimensions include:Usernames: requests from users with an exempt authenticated (or impersonated) username are ignored.RuntimeClassNames: pods and workload resources specifying an exempt runtime class name are ignored.Namespaces: pods and workload resources in an exempt namespace are ignored.Caution:Most pods are created by a controller in response to a workload resource, meaning that exempting an end user will only exempt them from enforcement when creating pods directly, but not when creating a workload resource. Controller service accounts (such as system:serviceaccount:kube-system:replicaset-controller) should generally not be exempted, as doing so would implicitly exempt any user that can create the corresponding workload resource.Updates to the following pod fields are exempt from policy checks, meaning that if a pod update request only changes these fields, it will not be denied even if the pod is in violation of the current policy level:Any metadata updates except changes to the seccomp or AppArmor annotations:seccomp.security.alpha.kubernetes.io/pod (deprecated)container.seccomp.security.alpha.kubernetes.io/* (deprecated)container.apparmor.security.beta.kubernetes.io/* (deprecated)Valid updates to .spec.activeDeadlineSecondsValid updates to .spec.tolerationsMetricsHere are the Prometheus metrics exposed by kube-apiserver:pod_security_errors_total: This metric indicates the number of errors preventing normal evaluation. Non-fatal errors may result in the latest restricted profile being used for enforcement.pod_security_evaluations_total: This metric indicates the number of policy evaluations that have occurred, not counting ignored or exempt requests during exporting.pod_security_exemptions_total: This metric indicates the number of exempt requests, not counting ignored or out of scope requests.What's nextPod Security StandardsEnforcing Pod Security StandardsEnforce Pod Security Standards by Configuring the Built-in Admission ControllerEnforce Pod Security Standards with Namespace LabelsIf you are running an older version of Kubernetes and want to upgrade to a version of Kubernetes that does not include PodSecurityPolicies, read migrate from PodSecurityPolicy to the Built-In PodSecurity Admission Controller.


# CSIDriver

CSIDriverCSIDriver captures information about a Container Storage Interface (CSI) volume driver deployed on the cluster.apiVersion: storage.k8s.io/v1import "k8s.io/api/storage/v1"CSIDriverCSIDriver captures information about a Container Storage Interface (CSI) volume driver deployed on the cluster. Kubernetes attach detach controller uses this object to determine whether attach is required. Kubelet uses this object to determine whether pod information needs to be passed on mount. CSIDriver objects are non-namespaced.apiVersion: storage.k8s.io/v1kind: CSIDrivermetadata (ObjectMeta)Standard object metadata. metadata.Name indicates the name of the CSI driver that this object refers to; it MUST be the same name returned by the CSI GetPluginName() call for that driver. The driver name must be 63 characters or less, beginning and ending with an alphanumeric character ([a-z0-9A-Z]) with dashes (-), dots (.), and alphanumerics between. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadataspec (CSIDriverSpec), requiredspec represents the specification of the CSI Driver.CSIDriverSpecCSIDriverSpec is the specification of a CSIDriver.attachRequired (boolean)attachRequired indicates this CSI volume driver requires an attach operation (because it implements the CSI ControllerPublishVolume() method), and that the Kubernetes attach detach controller should call the attach volume interface which checks the volumeattachment status and waits until the volume is attached before proceeding to mounting. The CSI external-attacher coordinates with CSI volume driver and updates the volumeattachment status when the attach operation is complete. If the value is specified to false, the attach operation will be skipped. Otherwise the attach operation will be called.This field is immutable.fsGroupPolicy (string)fsGroupPolicy defines if the underlying volume supports changing ownership and permission of the volume before being mounted. Refer to the specific FSGroupPolicy values for additional details.This field was immutable in Kubernetes < 1.29 and now is mutable.Defaults to ReadWriteOnceWithFSType, which will examine each volume to determine if Kubernetes should modify ownership and permissions of the volume. With the default policy the defined fsGroup will only be applied if a fstype is defined and the volume's access mode contains ReadWriteOnce.nodeAllocatableUpdatePeriodSeconds (int64)nodeAllocatableUpdatePeriodSeconds specifies the interval between periodic updates of the CSINode allocatable capacity for this driver. When set, both periodic updates and updates triggered by capacity-related failures are enabled. If not set, no updates occur (neither periodic nor upon detecting capacity-related failures), and the allocatable.count remains static. The minimum allowed value for this field is 10 seconds.This is a beta feature and requires the MutableCSINodeAllocatableCount feature gate to be enabled.This field is mutable.podInfoOnMount (boolean)podInfoOnMount indicates this CSI volume driver requires additional pod information (like podName, podUID, etc.) during mount operations, if set to true. If set to false, pod information will not be passed on mount. Default is false.The CSI driver specifies podInfoOnMount as part of driver deployment. If true, Kubelet will pass pod information as VolumeContext in the CSI NodePublishVolume() calls. The CSI driver is responsible for parsing and validating the information passed in as VolumeContext.The following VolumeContext will be passed if podInfoOnMount is set to true. This list might grow, but the prefix will be used. "csi.storage.k8s.io/pod.name": pod.Name "csi.storage.k8s.io/pod.namespace": pod.Namespace "csi.storage.k8s.io/pod.uid": string(pod.UID) "csi.storage.k8s.io/ephemeral": "true" if the volume is an ephemeral inline volume defined by a CSIVolumeSource, otherwise "false""csi.storage.k8s.io/ephemeral" is a new feature in Kubernetes 1.16. It is only required for drivers which support both the "Persistent" and "Ephemeral" VolumeLifecycleMode. Other drivers can leave pod info disabled and/or ignore this field. As Kubernetes 1.15 doesn't support this field, drivers can only support one mode when deployed on such a cluster and the deployment determines which mode that is, for example via a command line parameter of the driver.This field was immutable in Kubernetes < 1.29 and now is mutable.requiresRepublish (boolean)requiresRepublish indicates the CSI driver wants NodePublishVolume being periodically called to reflect any possible change in the mounted volume. This field defaults to false.Note: After a successful initial NodePublishVolume call, subsequent calls to NodePublishVolume should only update the contents of the volume. New mount points will not be seen by a running container.seLinuxMount (boolean)seLinuxMount specifies if the CSI driver supports "-o context" mount option.When "true", the CSI driver must ensure that all volumes provided by this CSI driver can be mounted separately with different -o context options. This is typical for storage backends that provide volumes as filesystems on block devices or as independent shared volumes. Kubernetes will call NodeStage / NodePublish with "-o context=xyz" mount option when mounting a ReadWriteOncePod volume used in Pod that has explicitly set SELinux context. In the future, it may be expanded to other volume AccessModes. In any case, Kubernetes will ensure that the volume is mounted only with a single SELinux context.When "false", Kubernetes won't pass any special SELinux mount options to the driver. This is typical for volumes that represent subdirectories of a bigger shared filesystem.Default is "false".storageCapacity (boolean)storageCapacity indicates that the CSI volume driver wants pod scheduling to consider the storage capacity that the driver deployment will report by creating CSIStorageCapacity objects with capacity information, if set to true.The check can be enabled immediately when deploying a driver. In that case, provisioning new volumes with late binding will pause until the driver deployment has published some suitable CSIStorageCapacity object.Alternatively, the driver can be deployed with the field unset or false and it can be flipped later when storage capacity information has been published.This field was immutable in Kubernetes <= 1.22 and now is mutable.tokenRequests ([]TokenRequest)Atomic: will be replaced during a mergetokenRequests indicates the CSI driver needs pods' service account tokens it is mounting volume for to do necessary authentication. Kubelet will pass the tokens in VolumeContext in the CSI NodePublishVolume calls. The CSI driver should parse and validate the following VolumeContext: "csi.storage.k8s.io/serviceAccount.tokens": { "<audience>": { "token": <token>, "expirationTimestamp": <expiration timestamp in RFC3339>, }, ... }Note: Audience in each TokenRequest should be different and at most one token is empty string. To receive a new token after expiry, RequiresRepublish can be used to trigger NodePublishVolume periodically.TokenRequest contains parameters of a service account token.tokenRequests.audience (string), requiredaudience is the intended audience of the token in "TokenRequestSpec". It will default to the audiences of kube apiserver.tokenRequests.expirationSeconds (int64)expirationSeconds is the duration of validity of the token in "TokenRequestSpec". It has the same default value of "ExpirationSeconds" in "TokenRequestSpec".volumeLifecycleModes ([]string)Set: unique values will be kept during a mergevolumeLifecycleModes defines what kind of volumes this CSI volume driver supports. The default if the list is empty is "Persistent", which is the usage defined by the CSI specification and implemented in Kubernetes via the usual PV/PVC mechanism.The other mode is "Ephemeral". In this mode, volumes are defined inline inside the pod spec with CSIVolumeSource and their lifecycle is tied to the lifecycle of that pod. A driver has to be aware of this because it is only going to get a NodePublishVolume call for such a volume.For more information about implementing this mode, see https://kubernetes-csi.github.io/docs/ephemeral-local-volumes.html A driver can support one or more of these modes and more modes may be added in the future.This field is beta. This field is immutable.CSIDriverListCSIDriverList is a collection of CSIDriver objects.apiVersion: storage.k8s.io/v1kind: CSIDriverListmetadata (ListMeta)Standard list metadata More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadataitems ([]CSIDriver), requireditems is the list of CSIDriverOperationsget read the specified CSIDriverHTTP RequestGET /apis/storage.k8s.io/v1/csidrivers/{name}Parametersname (in path): string, requiredname of the CSIDriverpretty (in query): stringprettyResponse200 (CSIDriver): OK401: Unauthorizedlist list or watch objects of kind CSIDriverHTTP RequestGET /apis/storage.k8s.io/v1/csidriversParametersallowWatchBookmarks (in query): booleanallowWatchBookmarkscontinue (in query): stringcontinuefieldSelector (in query): stringfieldSelectorlabelSelector (in query): stringlabelSelectorlimit (in query): integerlimitpretty (in query): stringprettyresourceVersion (in query): stringresourceVersionresourceVersionMatch (in query): stringresourceVersionMatchsendInitialEvents (in query): booleansendInitialEventstimeoutSeconds (in query): integertimeoutSecondswatch (in query): booleanwatchResponse200 (CSIDriverList): OK401: Unauthorizedcreate create a CSIDriverHTTP RequestPOST /apis/storage.k8s.io/v1/csidriversParametersbody: CSIDriver, requireddryRun (in query): stringdryRunfieldManager (in query): stringfieldManagerfieldValidation (in query): stringfieldValidationpretty (in query): stringprettyResponse200 (CSIDriver): OK201 (CSIDriver): Created202 (CSIDriver): Accepted401: Unauthorizedupdate replace the specified CSIDriverHTTP RequestPUT /apis/storage.k8s.io/v1/csidrivers/{name}Parametersname (in path): string, requiredname of the CSIDriverbody: CSIDriver, requireddryRun (in query): stringdryRunfieldManager (in query): stringfieldManagerfieldValidation (in query): stringfieldValidationpretty (in query): stringprettyResponse200 (CSIDriver): OK201 (CSIDriver): Created401: Unauthorizedpatch partially update the specified CSIDriverHTTP RequestPATCH /apis/storage.k8s.io/v1/csidrivers/{name}Parametersname (in path): string, requiredname of the CSIDriverbody: Patch, requireddryRun (in query): stringdryRunfieldManager (in query): stringfieldManagerfieldValidation (in query): stringfieldValidationforce (in query): booleanforcepretty (in query): stringprettyResponse200 (CSIDriver): OK201 (CSIDriver): Created401: Unauthorizeddelete delete a CSIDriverHTTP RequestDELETE /apis/storage.k8s.io/v1/csidrivers/{name}Parametersname (in path): string, requiredname of the CSIDriverbody: DeleteOptionsdryRun (in query): stringdryRungracePeriodSeconds (in query): integergracePeriodSecondsignoreStoreReadErrorWithClusterBreakingPotential (in query): booleanignoreStoreReadErrorWithClusterBreakingPotentialpretty (in query): stringprettypropagationPolicy (in query): stringpropagationPolicyResponse200 (CSIDriver): OK202 (CSIDriver): Accepted401: Unauthorizeddeletecollection delete collection of CSIDriverHTTP RequestDELETE /apis/storage.k8s.io/v1/csidriversParametersbody: DeleteOptionscontinue (in query): stringcontinuedryRun (in query): stringdryRunfieldSelector (in query): stringfieldSelectorgracePeriodSeconds (in query): integergracePeriodSecondsignoreStoreReadErrorWithClusterBreakingPotential (in query): booleanignoreStoreReadErrorWithClusterBreakingPotentiallabelSelector (in query): stringlabelSelectorlimit (in query): integerlimitpretty (in query): stringprettypropagationPolicy (in query): stringpropagationPolicyresourceVersion (in query): stringresourceVersionresourceVersionMatch (in query): stringresourceVersionMatchsendInitialEvents (in query): booleansendInitialEventstimeoutSeconds (in query): integertimeoutSecondsResponse200 (Status): OK401: UnauthorizedThis page is automatically generated.If you plan to report an issue with this page, mention that the page is auto-generated in your issue description. The fix may need to happen elsewhere in the Kubernetes project.


# Use a SOCKS5 Proxy to Access the Kubernetes API

Use a SOCKS5 Proxy to Access the Kubernetes APIFEATURE STATE: Kubernetes v1.24 [stable]This page shows how to use a SOCKS5 proxy to access the API of a remote Kubernetes cluster. This is useful when the cluster you want to access does not expose its API directly on the public internet.Before you beginYou need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. It is recommended to run this tutorial on a cluster with at least two nodes that are not acting as control plane hosts. If you do not already have a cluster, you can create one by using minikube or you can use one of these Kubernetes playgrounds:iximiuz LabsKillercodaKodeKloudPlay with KubernetesYour Kubernetes server must be at or later than version v1.24.To check the version, enter kubectl version.You need SSH client software (the ssh tool), and an SSH service running on the remote server. You must be able to log in to the SSH service on the remote server.Task contextNote:This example tunnels traffic using SSH, with the SSH client and server acting as a SOCKS proxy. You can instead use any other kind of SOCKS5 proxies.Figure 1 represents what you're going to achieve in this task.You have a client computer, referred to as local in the steps ahead, from where you're going to create requests to talk to the Kubernetes API.The Kubernetes server/API is hosted on a remote server.You will use SSH client and server software to create a secure SOCKS5 tunnel between the local and the remote server. The HTTPS traffic between the client and the Kubernetes API will flow over the SOCKS5 tunnel, which is itself tunnelled over SSH.graph LR; subgraph local[Local client machine] client([client])-. localtraffic .-> local_ssh[Local SSHSOCKS5 proxy]; end local_ssh[SSHSOCKS5proxy]-- SSH Tunnel -->sshd subgraph remote[Remote server] sshd[SSHserver]-- local traffic -->service1; end client([client])-. proxied HTTPs trafficgoing through the proxy .->service1[Kubernetes API]; classDef plain fill:#ddd,stroke:#fff,stroke-width:4px,color:#000; classDef k8s fill:#326ce5,stroke:#fff,stroke-width:4px,color:#fff; classDef cluster fill:#fff,stroke:#bbb,stroke-width:2px,color:#326ce5; class ingress,service1,service2,pod1,pod2,pod3,pod4 k8s; class client plain; class cluster cluster;JavaScript must be enabled to view this contentFigure 1. SOCKS5 tutorial componentsUsing ssh to create a SOCKS5 proxyThe following command starts a SOCKS5 proxy between your client machine and the remote SOCKS server:# The SSH tunnel continues running in the foreground after you run this ssh -D 1080 -q -N username@kubernetes-remote-server.example The SOCKS5 proxy lets you connect to your cluster's API server based on the following configuration:-D 1080: opens a SOCKS proxy on local port :1080.-q: quiet mode. Causes most warning and diagnostic messages to be suppressed.-N: Do not execute a remote command. Useful for just forwarding ports.username@kubernetes-remote-server.example: the remote SSH server behind which the Kubernetes cluster is running (eg: a bastion host).Client configurationTo access the Kubernetes API server through the proxy you must instruct kubectl to send queries through the SOCKS proxy we created earlier. Do this by either setting the appropriate environment variable, or via the proxy-url attribute in the kubeconfig file. Using an environment variable:export HTTPS_PROXY=socks5://localhost:1080 To always use this setting on a specific kubectl context, specify the proxy-url attribute in the relevant cluster entry within the ~/.kube/config file. For example:apiVersion: v1 clusters: - cluster: certificate-authority-data: LRMEMMW2 # shortened for readability server: https://<API_SERVER_IP_ADDRESS>:6443 # the "Kubernetes API" server, in other words the IP address of kubernetes-remote-server.example proxy-url: socks5://localhost:1080 # the "SSH SOCKS5 proxy" in the diagram above name: default contexts: - context: cluster: default user: default name: default current-context: default kind: Config preferences: {} users: - name: default user: client-certificate-data: LS0tLS1CR== # shortened for readability client-key-data: LS0tLS1CRUdJT= # shortened for readability Once you have created the tunnel via the ssh command mentioned earlier, and defined either the environment variable or the proxy-url attribute, you can interact with your cluster through that proxy. For example:kubectl get pods NAMESPACE NAME READY STATUS RESTARTS AGE kube-system coredns-85cb69466-klwq8 1/1 Running 0 5m46s Note:Before kubectl 1.24, most kubectl commands worked when using a socks proxy, except kubectl exec.kubectl supports both HTTPS_PROXY and https_proxy environment variables. These are used by other programs that support SOCKS, such as curl. Therefore in some cases it will be better to define the environment variable on the command line:HTTPS_PROXY=socks5://localhost:1080 kubectl get pods When using proxy-url, the proxy is used only for the relevant kubectl context, whereas the environment variable will affect all contexts.The k8s API server hostname can be further protected from DNS leakage by using the socks5h protocol name instead of the more commonly known socks5 protocol shown above. In this case, kubectl will ask the proxy server (such as an ssh bastion) to resolve the k8s API server domain name, instead of resolving it on the system running kubectl. Note also that with socks5h, a k8s API server URL like https://localhost:6443/api does not refer to your local client computer. Instead, it refers to localhost as known on the proxy server (eg the ssh bastion).Clean upStop the ssh port-forwarding process by pressing CTRL+C on the terminal where it is running.Type unset https_proxy in a terminal to stop forwarding http traffic through the proxy.Further readingOpenSSH remote login client


# Authorization Resources

Authorization ResourcesLocalSubjectAccessReviewLocalSubjectAccessReview checks whether or not a user or group can perform an action in a given namespace.SelfSubjectAccessReviewSelfSubjectAccessReview checks whether or the current user can perform an action.SelfSubjectRulesReviewSelfSubjectRulesReview enumerates the set of actions the current user can perform within a namespace.SubjectAccessReviewSubjectAccessReview checks whether or not a user or group can perform an action.ClusterRoleClusterRole is a cluster level, logical grouping of PolicyRules that can be referenced as a unit by a RoleBinding or ClusterRoleBinding.ClusterRoleBindingClusterRoleBinding references a ClusterRole, but not contain it.RoleRole is a namespaced, logical grouping of PolicyRules that can be referenced as a unit by a RoleBinding.RoleBindingRoleBinding references a role, but does not contain it.This page is automatically generated.If you plan to report an issue with this page, mention that the page is auto-generated in your issue description. The fix may need to happen elsewhere in the Kubernetes project.


# ResourceQuota

ResourceQuotaResourceQuota sets aggregate quota restrictions enforced per namespace.apiVersion: v1import "k8s.io/api/core/v1"ResourceQuotaResourceQuota sets aggregate quota restrictions enforced per namespaceapiVersion: v1kind: ResourceQuotametadata (ObjectMeta)Standard object's metadata. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadataspec (ResourceQuotaSpec)Spec defines the desired quota. https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-statusstatus (ResourceQuotaStatus)Status defines the actual enforced quota and its current usage. https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-statusResourceQuotaSpecResourceQuotaSpec defines the desired hard limits to enforce for Quota.hard (map[string]Quantity)hard is the set of desired hard limits for each named resource. More info: https://kubernetes.io/docs/concepts/policy/resource-quotas/scopeSelector (ScopeSelector)scopeSelector is also a collection of filters like scopes that must match each object tracked by a quota but expressed using ScopeSelectorOperator in combination with possible values. For a resource to match, both scopes AND scopeSelector (if specified in spec), must be matched.A scope selector represents the AND of the selectors represented by the scoped-resource selector requirements.scopeSelector.matchExpressions ([]ScopedResourceSelectorRequirement)Atomic: will be replaced during a mergeA list of scope selector requirements by scope of the resources.A scoped-resource selector requirement is a selector that contains values, a scope name, and an operator that relates the scope name and values.scopeSelector.matchExpressions.operator (string), requiredRepresents a scope's relationship to a set of values. Valid operators are In, NotIn, Exists, DoesNotExist.scopeSelector.matchExpressions.scopeName (string), requiredThe name of the scope that the selector applies to.scopeSelector.matchExpressions.values ([]string)Atomic: will be replaced during a mergeAn array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. This array is replaced during a strategic merge patch.scopes ([]string)Atomic: will be replaced during a mergeA collection of filters that must match each object tracked by a quota. If not specified, the quota matches all objects.ResourceQuotaStatusResourceQuotaStatus defines the enforced hard limits and observed use.hard (map[string]Quantity)Hard is the set of enforced hard limits for each named resource. More info: https://kubernetes.io/docs/concepts/policy/resource-quotas/used (map[string]Quantity)Used is the current observed total usage of the resource in the namespace.ResourceQuotaListResourceQuotaList is a list of ResourceQuota items.apiVersion: v1kind: ResourceQuotaListmetadata (ListMeta)Standard list metadata. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kindsitems ([]ResourceQuota), requiredItems is a list of ResourceQuota objects. More info: https://kubernetes.io/docs/concepts/policy/resource-quotas/Operationsget read the specified ResourceQuotaHTTP RequestGET /api/v1/namespaces/{namespace}/resourcequotas/{name}Parametersname (in path): string, requiredname of the ResourceQuotanamespace (in path): string, requirednamespacepretty (in query): stringprettyResponse200 (ResourceQuota): OK401: Unauthorizedget read status of the specified ResourceQuotaHTTP RequestGET /api/v1/namespaces/{namespace}/resourcequotas/{name}/statusParametersname (in path): string, requiredname of the ResourceQuotanamespace (in path): string, requirednamespacepretty (in query): stringprettyResponse200 (ResourceQuota): OK401: Unauthorizedlist list or watch objects of kind ResourceQuotaHTTP RequestGET /api/v1/namespaces/{namespace}/resourcequotasParametersnamespace (in path): string, requirednamespaceallowWatchBookmarks (in query): booleanallowWatchBookmarkscontinue (in query): stringcontinuefieldSelector (in query): stringfieldSelectorlabelSelector (in query): stringlabelSelectorlimit (in query): integerlimitpretty (in query): stringprettyresourceVersion (in query): stringresourceVersionresourceVersionMatch (in query): stringresourceVersionMatchsendInitialEvents (in query): booleansendInitialEventstimeoutSeconds (in query): integertimeoutSecondswatch (in query): booleanwatchResponse200 (ResourceQuotaList): OK401: Unauthorizedlist list or watch objects of kind ResourceQuotaHTTP RequestGET /api/v1/resourcequotasParametersallowWatchBookmarks (in query): booleanallowWatchBookmarkscontinue (in query): stringcontinuefieldSelector (in query): stringfieldSelectorlabelSelector (in query): stringlabelSelectorlimit (in query): integerlimitpretty (in query): stringprettyresourceVersion (in query): stringresourceVersionresourceVersionMatch (in query): stringresourceVersionMatchsendInitialEvents (in query): booleansendInitialEventstimeoutSeconds (in query): integertimeoutSecondswatch (in query): booleanwatchResponse200 (ResourceQuotaList): OK401: Unauthorizedcreate create a ResourceQuotaHTTP RequestPOST /api/v1/namespaces/{namespace}/resourcequotasParametersnamespace (in path): string, requirednamespacebody: ResourceQuota, requireddryRun (in query): stringdryRunfieldManager (in query): stringfieldManagerfieldValidation (in query): stringfieldValidationpretty (in query): stringprettyResponse200 (ResourceQuota): OK201 (ResourceQuota): Created202 (ResourceQuota): Accepted401: Unauthorizedupdate replace the specified ResourceQuotaHTTP RequestPUT /api/v1/namespaces/{namespace}/resourcequotas/{name}Parametersname (in path): string, requiredname of the ResourceQuotanamespace (in path): string, requirednamespacebody: ResourceQuota, requireddryRun (in query): stringdryRunfieldManager (in query): stringfieldManagerfieldValidation (in query): stringfieldValidationpretty (in query): stringprettyResponse200 (ResourceQuota): OK201 (ResourceQuota): Created401: Unauthorizedupdate replace status of the specified ResourceQuotaHTTP RequestPUT /api/v1/namespaces/{namespace}/resourcequotas/{name}/statusParametersname (in path): string, requiredname of the ResourceQuotanamespace (in path): string, requirednamespacebody: ResourceQuota, requireddryRun (in query): stringdryRunfieldManager (in query): stringfieldManagerfieldValidation (in query): stringfieldValidationpretty (in query): stringprettyResponse200 (ResourceQuota): OK201 (ResourceQuota): Created401: Unauthorizedpatch partially update the specified ResourceQuotaHTTP RequestPATCH /api/v1/namespaces/{namespace}/resourcequotas/{name}Parametersname (in path): string, requiredname of the ResourceQuotanamespace (in path): string, requirednamespacebody: Patch, requireddryRun (in query): stringdryRunfieldManager (in query): stringfieldManagerfieldValidation (in query): stringfieldValidationforce (in query): booleanforcepretty (in query): stringprettyResponse200 (ResourceQuota): OK201 (ResourceQuota): Created401: Unauthorizedpatch partially update status of the specified ResourceQuotaHTTP RequestPATCH /api/v1/namespaces/{namespace}/resourcequotas/{name}/statusParametersname (in path): string, requiredname of the ResourceQuotanamespace (in path): string, requirednamespacebody: Patch, requireddryRun (in query): stringdryRunfieldManager (in query): stringfieldManagerfieldValidation (in query): stringfieldValidationforce (in query): booleanforcepretty (in query): stringprettyResponse200 (ResourceQuota): OK201 (ResourceQuota): Created401: Unauthorizeddelete delete a ResourceQuotaHTTP RequestDELETE /api/v1/namespaces/{namespace}/resourcequotas/{name}Parametersname (in path): string, requiredname of the ResourceQuotanamespace (in path): string, requirednamespacebody: DeleteOptionsdryRun (in query): stringdryRungracePeriodSeconds (in query): integergracePeriodSecondsignoreStoreReadErrorWithClusterBreakingPotential (in query): booleanignoreStoreReadErrorWithClusterBreakingPotentialpretty (in query): stringprettypropagationPolicy (in query): stringpropagationPolicyResponse200 (ResourceQuota): OK202 (ResourceQuota): Accepted401: Unauthorizeddeletecollection delete collection of ResourceQuotaHTTP RequestDELETE /api/v1/namespaces/{namespace}/resourcequotasParametersnamespace (in path): string, requirednamespacebody: DeleteOptionscontinue (in query): stringcontinuedryRun (in query): stringdryRunfieldSelector (in query): stringfieldSelectorgracePeriodSeconds (in query): integergracePeriodSecondsignoreStoreReadErrorWithClusterBreakingPotential (in query): booleanignoreStoreReadErrorWithClusterBreakingPotentiallabelSelector (in query): stringlabelSelectorlimit (in query): integerlimitpretty (in query): stringprettypropagationPolicy (in query): stringpropagationPolicyresourceVersion (in query): stringresourceVersionresourceVersionMatch (in query): stringresourceVersionMatchsendInitialEvents (in query): booleansendInitialEventstimeoutSeconds (in query): integertimeoutSecondsResponse200 (Status): OK401: UnauthorizedThis page is automatically generated.If you plan to report an issue with this page, mention that the page is auto-generated in your issue description. The fix may need to happen elsewhere in the Kubernetes project.


# Debug Pods

Debug PodsThis guide is to help users debug applications that are deployed into Kubernetes and not behaving correctly. This is not a guide for people who want to debug their cluster. For that you should check out this guide.Diagnosing the problemThe first step in troubleshooting is triage. What is the problem? Is it your Pods, your Replication Controller or your Service?Debugging PodsDebugging Replication ControllersDebugging ServicesDebugging PodsThe first step in debugging a Pod is taking a look at it. Check the current state of the Pod and recent events with the following command:kubectl describe pods ${POD_NAME} Look at the state of the containers in the pod. Are they all Running? Have there been recent restarts?Continue debugging depending on the state of the pods.My pod stays pendingIf a Pod is stuck in Pending it means that it can not be scheduled onto a node. Generally this is because there are insufficient resources of one type or another that prevent scheduling. Look at the output of the kubectl describe ... command above. There should be messages from the scheduler about why it can not schedule your pod. Reasons include:You don't have enough resources: You may have exhausted the supply of CPU or Memory in your cluster, in this case you need to delete Pods, adjust resource requests, or add new nodes to your cluster. See Compute Resources document for more information.You are using hostPort: When you bind a Pod to a hostPort there are a limited number of places that pod can be scheduled. In most cases, hostPort is unnecessary, try using a Service object to expose your Pod. If you do require hostPort then you can only schedule as many Pods as there are nodes in your Kubernetes cluster.My pod stays waitingIf a Pod is stuck in the Waiting state, then it has been scheduled to a worker node, but it can't run on that machine. Again, the information from kubectl describe ... should be informative. The most common cause of Waiting pods is a failure to pull the image. There are three things to check:Make sure that you have the name of the image correct.Have you pushed the image to the registry?Try to manually pull the image to see if the image can be pulled. For example, if you use Docker on your PC, run docker pull <image>.My pod stays terminatingIf a Pod is stuck in the Terminating state, it means that a deletion has been issued for the Pod, but the control plane is unable to delete the Pod object.This typically happens if the Pod has a finalizer and there is an admission webhook installed in the cluster that prevents the control plane from removing the finalizer.To identify this scenario, check if your cluster has any ValidatingWebhookConfiguration or MutatingWebhookConfiguration that target UPDATE operations for pods resources.If the webhook is provided by a third-party:Make sure you are using the latest version.Disable the webhook for UPDATE operations.Report an issue with the corresponding provider.If you are the author of the webhook:For a mutating webhook, make sure it never changes immutable fields on UPDATE operations. For example, changes to containers are usually not allowed.For a validating webhook, make sure that your validation policies only apply to new changes. In other words, you should allow Pods with existing violations to pass validation. This allows Pods that were created before the validating webhook was installed to continue running.My pod is crashing or otherwise unhealthyOnce your pod has been scheduled, the methods described in Debug Running Pods are available for debugging.My pod is running but not doing what I told it to doIf your pod is not behaving as you expected, it may be that there was an error in your pod description (e.g. mypod.yaml file on your local machine), and that the error was silently ignored when you created the pod. Often a section of the pod description is nested incorrectly, or a key name is typed incorrectly, and so the key is ignored. For example, if you misspelled command as commnd then the pod will be created but will not use the command line you intended it to use.The first thing to do is to delete your pod and try creating it again with the --validate option. For example, run kubectl apply --validate -f mypod.yaml. If you misspelled command as commnd then will give an error like this:I0805 10:43:25.129850 46757 schema.go:126] unknown field: commnd I0805 10:43:25.129973 46757 schema.go:129] this may be a false alarm, see https://github.com/kubernetes/kubernetes/issues/6842 pods/mypod The next thing to check is whether the pod on the apiserver matches the pod you meant to create (e.g. in a yaml file on your local machine). For example, run kubectl get pods/mypod -o yaml > mypod-on-apiserver.yaml and then manually compare the original pod description, mypod.yaml with the one you got back from apiserver, mypod-on-apiserver.yaml. There will typically be some lines on the "apiserver" version that are not on the original version. This is expected. However, if there are lines on the original that are not on the apiserver version, then this may indicate a problem with your pod spec.Debugging Replication ControllersReplication controllers are fairly straightforward. They can either create Pods or they can't. If they can't create pods, then please refer to the instructions above to debug your pods.You can also use kubectl describe rc ${CONTROLLER_NAME} to introspect events related to the replication controller.Debugging ServicesServices provide load balancing across a set of pods. There are several common problems that can make Services not work properly. The following instructions should help debug Service problems.First, verify that there are endpoints for the service. For every Service object, the apiserver makes one or more EndpointSlice resources available.You can view these resources with:kubectl get endpointslices -l kubernetes.io/service-name=${SERVICE_NAME} Make sure that the endpoints in the EndpointSlices match up with the number of pods that you expect to be members of your service. For example, if your Service is for an nginx container with 3 replicas, you would expect to see three different IP addresses in the Service's endpoint slices.My service is missing endpointsIf you are missing endpoints, try listing pods using the labels that Service uses. Imagine that you have a Service where the labels are:... spec: - selector: name: nginx type: frontend You can use:kubectl get pods --selector=name=nginx,type=frontend to list pods that match this selector. Verify that the list matches the Pods that you expect to provide your Service. Verify that the pod's containerPort matches up with the Service's targetPortNetwork traffic is not forwardedPlease see debugging service for more information.What's nextIf none of the above solves your problem, follow the instructions in Debugging Service document to make sure that your Service is running, has Endpoints, and your Pods are actually serving; you have DNS working, iptables rules installed, and kube-proxy does not seem to be misbehaving.You may also visit troubleshooting document for more information.


# Assigning Pods to Nodes

Assigning Pods to NodesYou can constrain a Pod so that it is restricted to run on particular node(s), or to prefer to run on particular nodes. There are several ways to do this and the recommended approaches all use label selectors to facilitate the selection. Often, you do not need to set any such constraints; the scheduler will automatically do a reasonable placement (for example, spreading your Pods across nodes so as not place Pods on a node with insufficient free resources). However, there are some circumstances where you may want to control which node the Pod deploys to, for example, to ensure that a Pod ends up on a node with an SSD attached to it, or to co-locate Pods from two different services that communicate a lot into the same availability zone.You can use any of the following methods to choose where Kubernetes schedules specific Pods:nodeSelector field matching against node labelsAffinity and anti-affinitynodeName fieldPod topology spread constraintsNode labelsLike many other Kubernetes objects, nodes have labels. You can attach labels manually. Kubernetes also populates a standard set of labels on all nodes in a cluster.Note:The value of these labels is cloud provider specific and is not guaranteed to be reliable. For example, the value of kubernetes.io/hostname may be the same as the node name in some environments and a different value in other environments.Node isolation/restrictionAdding labels to nodes allows you to target Pods for scheduling on specific nodes or groups of nodes. You can use this functionality to ensure that specific Pods only run on nodes with certain isolation, security, or regulatory properties.If you use labels for node isolation, choose label keys that the kubelet cannot modify. This prevents a compromised node from setting those labels on itself so that the scheduler schedules workloads onto the compromised node.The NodeRestriction admission plugin prevents the kubelet from setting or modifying labels with a node-restriction.kubernetes.io/ prefix.To make use of that label prefix for node isolation:Ensure you are using the Node authorizer and have enabled the NodeRestriction admission plugin.Add labels with the node-restriction.kubernetes.io/ prefix to your nodes, and use those labels in your node selectors. For example, example.com.node-restriction.kubernetes.io/fips=true or example.com.node-restriction.kubernetes.io/pci-dss=true.nodeSelectornodeSelector is the simplest recommended form of node selection constraint. You can add the nodeSelector field to your Pod specification and specify the node labels you want the target node to have. Kubernetes only schedules the Pod onto nodes that have each of the labels you specify.See Assign Pods to Nodes for more information.Affinity and anti-affinitynodeSelector is the simplest way to constrain Pods to nodes with specific labels. Affinity and anti-affinity expand the types of constraints you can define. Some of the benefits of affinity and anti-affinity include:The affinity/anti-affinity language is more expressive. nodeSelector only selects nodes with all the specified labels. Affinity/anti-affinity gives you more control over the selection logic.You can indicate that a rule is soft or preferred, so that the scheduler still schedules the Pod even if it can't find a matching node.You can constrain a Pod using labels on other Pods running on the node (or other topological domain), instead of just node labels, which allows you to define rules for which Pods can be co-located on a node.The affinity feature consists of two types of affinity:Node affinity functions like the nodeSelector field but is more expressive and allows you to specify soft rules.Inter-pod affinity/anti-affinity allows you to constrain Pods against labels on other Pods.Node affinityNode affinity is conceptually similar to nodeSelector, allowing you to constrain which nodes your Pod can be scheduled on based on node labels. There are two types of node affinity:requiredDuringSchedulingIgnoredDuringExecution: The scheduler can't schedule the Pod unless the rule is met. This functions like nodeSelector, but with a more expressive syntax.preferredDuringSchedulingIgnoredDuringExecution: The scheduler tries to find a node that meets the rule. If a matching node is not available, the scheduler still schedules the Pod.Note:In the preceding types, IgnoredDuringExecution means that if the node labels change after Kubernetes schedules the Pod, the Pod continues to run.You can specify node affinities using the .spec.affinity.nodeAffinity field in your Pod spec.For example, consider the following Pod spec:pods/pod-with-node-affinity.yaml apiVersion: v1 kind: Pod metadata: name: with-node-affinity spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: topology.kubernetes.io/zone operator: In values: - antarctica-east1 - antarctica-west1 preferredDuringSchedulingIgnoredDuringExecution: - weight: 1 preference: matchExpressions: - key: another-node-label-key operator: In values: - another-node-label-value containers: - name: with-node-affinity image: registry.k8s.io/pause:3.8In this example, the following rules apply:The node must have a label with the key topology.kubernetes.io/zone and the value of that label must be either antarctica-east1 or antarctica-west1.The node preferably has a label with the key another-node-label-key and the value another-node-label-value.You can use the operator field to specify a logical operator for Kubernetes to use when interpreting the rules. You can use In, NotIn, Exists, DoesNotExist, Gt and Lt.Read Operators to learn more about how these work.NotIn and DoesNotExist allow you to define node anti-affinity behavior. Alternatively, you can use node taints to repel Pods from specific nodes.Note:If you specify both nodeSelector and nodeAffinity, both must be satisfied for the Pod to be scheduled onto a node.If you specify multiple terms in nodeSelectorTerms associated with nodeAffinity types, then the Pod can be scheduled onto a node if one of the specified terms can be satisfied (terms are ORed).If you specify multiple expressions in a single matchExpressions field associated with a term in nodeSelectorTerms, then the Pod can be scheduled onto a node only if all the expressions are satisfied (expressions are ANDed).See Assign Pods to Nodes using Node Affinity for more information.Node affinity weightYou can specify a weight between 1 and 100 for each instance of the preferredDuringSchedulingIgnoredDuringExecution affinity type. When the scheduler finds nodes that meet all the other scheduling requirements of the Pod, the scheduler iterates through every preferred rule that the node satisfies and adds the value of the weight for that expression to a sum.The final sum is added to the score of other priority functions for the node. Nodes with the highest total score are prioritized when the scheduler makes a scheduling decision for the Pod.For example, consider the following Pod spec:pods/pod-with-affinity-preferred-weight.yaml apiVersion: v1 kind: Pod metadata: name: with-affinity-preferred-weight spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/os operator: In values: - linux preferredDuringSchedulingIgnoredDuringExecution: - weight: 1 preference: matchExpressions: - key: label-1 operator: In values: - key-1 - weight: 50 preference: matchExpressions: - key: label-2 operator: In values: - key-2 containers: - name: with-node-affinity image: registry.k8s.io/pause:3.8 If there are two possible nodes that match the preferredDuringSchedulingIgnoredDuringExecution rule, one with the label-1:key-1 label and another with the label-2:key-2 label, the scheduler considers the weight of each node and adds the weight to the other scores for that node, and schedules the Pod onto the node with the highest final score.Note:If you want Kubernetes to successfully schedule the Pods in this example, you must have existing nodes with the kubernetes.io/os=linux label.Node affinity per scheduling profileFEATURE STATE: Kubernetes v1.20 [beta]When configuring multiple scheduling profiles, you can associate a profile with a node affinity, which is useful if a profile only applies to a specific set of nodes. To do so, add an addedAffinity to the args field of the NodeAffinity plugin in the scheduler configuration. For example:apiVersion: kubescheduler.config.k8s.io/v1 kind: KubeSchedulerConfiguration profiles: - schedulerName: default-scheduler - schedulerName: foo-scheduler pluginConfig: - name: NodeAffinity args: addedAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: scheduler-profile operator: In values: - foo The addedAffinity is applied to all Pods that set .spec.schedulerName to foo-scheduler, in addition to the NodeAffinity specified in the PodSpec. That is, in order to match the Pod, nodes need to satisfy addedAffinity and the Pod's .spec.NodeAffinity.Since the addedAffinity is not visible to end users, its behavior might be unexpected to them. Use node labels that have a clear correlation to the scheduler profile name.Note:The DaemonSet controller, which creates Pods for DaemonSets, does not support scheduling profiles. When the DaemonSet controller creates Pods, the default Kubernetes scheduler places those Pods and honors any nodeAffinity rules in the DaemonSet controller.Inter-pod affinity and anti-affinityInter-pod affinity and anti-affinity allow you to constrain which nodes your Pods can be scheduled on based on the labels of Pods already running on that node, instead of the node labels.Types of Inter-pod Affinity and Anti-affinityInter-pod affinity and anti-affinity take the form "this Pod should (or, in the case of anti-affinity, should not) run in an X if that X is already running one or more Pods that meet rule Y", where X is a topology domain like node, rack, cloud provider zone or region, or similar and Y is the rule Kubernetes tries to satisfy.You express these rules (Y) as label selectors with an optional associated list of namespaces. Pods are namespaced objects in Kubernetes, so Pod labels also implicitly have namespaces. Any label selectors for Pod labels should specify the namespaces in which Kubernetes should look for those labels.You express the topology domain (X) using a topologyKey, which is the key for the node label that the system uses to denote the domain. For examples, see Well-Known Labels, Annotations and Taints.Note:Inter-pod affinity and anti-affinity require substantial amounts of processing which can slow down scheduling in large clusters significantly. We do not recommend using them in clusters larger than several hundred nodes.Note:Pod anti-affinity requires nodes to be consistently labeled, in other words, every node in the cluster must have an appropriate label matching topologyKey. If some or all nodes are missing the specified topologyKey label, it can lead to unintended behavior.Similar to node affinity are two types of Pod affinity and anti-affinity as follows:requiredDuringSchedulingIgnoredDuringExecutionpreferredDuringSchedulingIgnoredDuringExecutionFor example, you could use requiredDuringSchedulingIgnoredDuringExecution affinity to tell the scheduler to co-locate Pods of two services in the same cloud provider zone because they communicate with each other a lot. Similarly, you could use preferredDuringSchedulingIgnoredDuringExecution anti-affinity to spread Pods from a service across multiple cloud provider zones.To use inter-pod affinity, use the affinity.podAffinity field in the Pod spec. For inter-pod anti-affinity, use the affinity.podAntiAffinity field in the Pod spec.Scheduling BehaviorWhen scheduling a new Pod, the Kubernetes scheduler evaluates the Pod's affinity/anti-affinity rules in the context of the current cluster state:Hard Constraints (Node Filtering):podAffinity.requiredDuringSchedulingIgnoredDuringExecution and podAntiAffinity.requiredDuringSchedulingIgnoredDuringExecution:The scheduler ensures the new Pod is assigned to nodes that satisfy these required affinity and anti-affinity rules based on existing Pods.Soft Constraints (Scoring):podAffinity.preferredDuringSchedulingIgnoredDuringExecution and podAntiAffinity.preferredDuringSchedulingIgnoredDuringExecution:The scheduler scores nodes based on how well they meet these preferred affinity and anti-affinity rules to optimize Pod placement.Ignored Fields:Existing Pods' podAffinity.preferredDuringSchedulingIgnoredDuringExecution:These preferred affinity rules are not considered during the scheduling decision for new Pods.Existing Pods' podAntiAffinity.preferredDuringSchedulingIgnoredDuringExecution:Similarly, preferred anti-affinity rules of existing Pods are ignored during scheduling.Scheduling a Group of Pods with Inter-pod Affinity to ThemselvesIf the current Pod being scheduled is the first in a series that have affinity to themselves, it is allowed to be scheduled if it passes all other affinity checks. This is determined by verifying that no other Pod in the cluster matches the namespace and selector of this Pod, that the Pod matches its own terms, and the chosen node matches all requested topologies. This ensures that there will not be a deadlock even if all the Pods have inter-pod affinity specified.Pod Affinity ExampleConsider the following Pod spec:pods/pod-with-pod-affinity.yaml apiVersion: v1 kind: Pod metadata: name: with-pod-affinity spec: affinity: podAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchExpressions: - key: security operator: In values: - S1 topologyKey: topology.kubernetes.io/zone podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - weight: 100 podAffinityTerm: labelSelector: matchExpressions: - key: security operator: In values: - S2 topologyKey: topology.kubernetes.io/zone containers: - name: with-pod-affinity image: registry.k8s.io/pause:3.8 This example defines one Pod affinity rule and one Pod anti-affinity rule. The Pod affinity rule uses the "hard" requiredDuringSchedulingIgnoredDuringExecution, while the anti-affinity rule uses the "soft" preferredDuringSchedulingIgnoredDuringExecution.The affinity rule specifies that the scheduler is allowed to place the example Pod on a node only if that node belongs to a specific zone where other Pods have been labeled with security=S1. For instance, if we have a cluster with a designated zone, let's call it "Zone V," consisting of nodes labeled with topology.kubernetes.io/zone=V, the scheduler can assign the Pod to any node within Zone V, as long as there is at least one Pod within Zone V already labeled with security=S1. Conversely, if there are no Pods with security=S1 labels in Zone V, the scheduler will not assign the example Pod to any node in that zone.The anti-affinity rule specifies that the scheduler should try to avoid scheduling the Pod on a node if that node belongs to a specific zone where other Pods have been labeled with security=S2. For instance, if we have a cluster with a designated zone, let's call it "Zone R," consisting of nodes labeled with topology.kubernetes.io/zone=R, the scheduler should avoid assigning the Pod to any node within Zone R, as long as there is at least one Pod within Zone R already labeled with security=S2. Conversely, the anti-affinity rule does not impact scheduling into Zone R if there are no Pods with security=S2 labels.To get yourself more familiar with the examples of Pod affinity and anti-affinity, refer to the design proposal.You can use the In, NotIn, Exists and DoesNotExist values in the operator field for Pod affinity and anti-affinity.Read Operators to learn more about how these work.In principle, the topologyKey can be any allowed label key with the following exceptions for performance and security reasons:For Pod affinity and anti-affinity, an empty topologyKey field is not allowed in both requiredDuringSchedulingIgnoredDuringExecution and preferredDuringSchedulingIgnoredDuringExecution.For requiredDuringSchedulingIgnoredDuringExecution Pod anti-affinity rules, the admission controller LimitPodHardAntiAffinityTopology limits topologyKey to kubernetes.io/hostname. You can modify or disable the admission controller if you want to allow custom topologies.In addition to labelSelector and topologyKey, you can optionally specify a list of namespaces which the labelSelector should match against using the namespaces field at the same level as labelSelector and topologyKey. If omitted or empty, namespaces defaults to the namespace of the Pod where the affinity/anti-affinity definition appears.Namespace SelectorFEATURE STATE: Kubernetes v1.24 [stable]You can also select matching namespaces using namespaceSelector, which is a label query over the set of namespaces. The affinity term is applied to namespaces selected by both namespaceSelector and the namespaces field. Note that an empty namespaceSelector ({}) matches all namespaces, while a null or empty namespaces list and null namespaceSelector matches the namespace of the Pod where the rule is defined.matchLabelKeysFEATURE STATE: Kubernetes v1.33 [stable] (enabled by default: true)Note:The matchLabelKeys field is a beta-level field and is enabled by default in Kubernetes 1.34. When you want to disable it, you have to disable it explicitly via the MatchLabelKeysInPodAffinity feature gate.Kubernetes includes an optional matchLabelKeys field for Pod affinity or anti-affinity. The field specifies keys for the labels that should match with the incoming Pod's labels, when satisfying the Pod (anti)affinity.The keys are used to look up values from the Pod labels; those key-value labels are combined (using AND) with the match restrictions defined using the labelSelector field. The combined filtering selects the set of existing Pods that will be taken into Pod (anti)affinity calculation.Caution:It's not recommended to use matchLabelKeys with labels that might be updated directly on pods. Even if you edit the pod's label that is specified at matchLabelKeys directly, (that is, not via a deployment), kube-apiserver doesn't reflect the label update onto the merged labelSelector.A common use case is to use matchLabelKeys with pod-template-hash (set on Pods managed as part of a Deployment, where the value is unique for each revision). Using pod-template-hash in matchLabelKeys allows you to target the Pods that belong to the same revision as the incoming Pod, so that a rolling upgrade won't break affinity.apiVersion: apps/v1 kind: Deployment metadata: name: application-server ... spec: template: spec: affinity: podAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchExpressions: - key: app operator: In values: - database topologyKey: topology.kubernetes.io/zone # Only Pods from a given rollout are taken into consideration when calculating pod affinity. # If you update the Deployment, the replacement Pods follow their own affinity rules # (if there are any defined in the new Pod template) matchLabelKeys: - pod-template-hash mismatchLabelKeysFEATURE STATE: Kubernetes v1.33 [stable] (enabled by default: true)Note:The mismatchLabelKeys field is a beta-level field and is enabled by default in Kubernetes 1.34. When you want to disable it, you have to disable it explicitly via the MatchLabelKeysInPodAffinity feature gate.Kubernetes includes an optional mismatchLabelKeys field for Pod affinity or anti-affinity. The field specifies keys for the labels that should not match with the incoming Pod's labels, when satisfying the Pod (anti)affinity.Caution:It's not recommended to use mismatchLabelKeys with labels that might be updated directly on pods. Even if you edit the pod's label that is specified at mismatchLabelKeys directly, (that is, not via a deployment), kube-apiserver doesn't reflect the label update onto the merged labelSelector.One example use case is to ensure Pods go to the topology domain (node, zone, etc) where only Pods from the same tenant or team are scheduled in. In other words, you want to avoid running Pods from two different tenants on the same topology domain at the same time.apiVersion: v1 kind: Pod metadata: labels: # Assume that all relevant Pods have a "tenant" label set tenant: tenant-a ... spec: affinity: podAffinity: requiredDuringSchedulingIgnoredDuringExecution: # ensure that Pods associated with this tenant land on the correct node pool - matchLabelKeys: - tenant topologyKey: node-pool podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: # ensure that Pods associated with this tenant can't schedule to nodes used for another tenant - mismatchLabelKeys: - tenant # whatever the value of the "tenant" label for this Pod, prevent # scheduling to nodes in any pool where any Pod from a different # tenant is running. labelSelector: # We have to have the labelSelector which selects only Pods with the tenant label, # otherwise this Pod would have anti-affinity against Pods from daemonsets as well, for example, # which aren't supposed to have the tenant label. matchExpressions: - key: tenant operator: Exists topologyKey: node-pool More practical use-casesInter-pod affinity and anti-affinity can be even more useful when they are used with higher level collections such as ReplicaSets, StatefulSets, Deployments, etc. These rules allow you to configure that a set of workloads should be co-located in the same defined topology; for example, preferring to place two related Pods onto the same node.For example: imagine a three-node cluster. You use the cluster to run a web application and also an in-memory cache (such as Redis). For this example, also assume that latency between the web application and the memory cache should be as low as is practical. You could use inter-pod affinity and anti-affinity to co-locate the web servers with the cache as much as possible.In the following example Deployment for the Redis cache, the replicas get the label app=store. The podAntiAffinity rule tells the scheduler to avoid placing multiple replicas with the app=store label on a single node. This creates each cache in a separate node.apiVersion: apps/v1 kind: Deployment metadata: name: redis-cache spec: selector: matchLabels: app: store replicas: 3 template: metadata: labels: app: store spec: affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchExpressions: - key: app operator: In values: - store topologyKey: "kubernetes.io/hostname" containers: - name: redis-server image: redis:3.2-alpine The following example Deployment for the web servers creates replicas with the label app=web-store. The Pod affinity rule tells the scheduler to place each replica on a node that has a Pod with the label app=store. The Pod anti-affinity rule tells the scheduler never to place multiple app=web-store servers on a single node.apiVersion: apps/v1 kind: Deployment metadata: name: web-server spec: selector: matchLabels: app: web-store replicas: 3 template: metadata: labels: app: web-store spec: affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchExpressions: - key: app operator: In values: - web-store topologyKey: "kubernetes.io/hostname" podAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchExpressions: - key: app operator: In values: - store topologyKey: "kubernetes.io/hostname" containers: - name: web-app image: nginx:1.16-alpine Creating the two preceding Deployments results in the following cluster layout, where each web server is co-located with a cache, on three separate nodes.node-1node-2node-3webserver-1webserver-2webserver-3cache-1cache-2cache-3The overall effect is that each cache instance is likely to be accessed by a single client that is running on the same node. This approach aims to minimize both skew (imbalanced load) and latency.You might have other reasons to use Pod anti-affinity. See the ZooKeeper tutorial for an example of a StatefulSet configured with anti-affinity for high availability, using the same technique as this example.nodeNamenodeName is a more direct form of node selection than affinity or nodeSelector. nodeName is a field in the Pod spec. If the nodeName field is not empty, the scheduler ignores the Pod and the kubelet on the named node tries to place the Pod on that node. Using nodeName overrules using nodeSelector or affinity and anti-affinity rules.Some of the limitations of using nodeName to select nodes are:If the named node does not exist, the Pod will not run, and in some cases may be automatically deleted.If the named node does not have the resources to accommodate the Pod, the Pod will fail and its reason will indicate why, for example OutOfmemory or OutOfcpu.Node names in cloud environments are not always predictable or stable.Warning:nodeName is intended for use by custom schedulers or advanced use cases where you need to bypass any configured schedulers. Bypassing the schedulers might lead to failed Pods if the assigned Nodes get oversubscribed. You can use node affinity or the nodeSelector field to assign a Pod to a specific Node without bypassing the schedulers.Here is an example of a Pod spec using the nodeName field:apiVersion: v1 kind: Pod metadata: name: nginx spec: containers: - name: nginx image: nginx nodeName: kube-01 The above Pod will only run on the node kube-01.nominatedNodeNameFEATURE STATE: Kubernetes v1.34 [alpha] (enabled by default: false)nominatedNodeName can be used for external components to nominate node for a pending pod. This nomination is best effort: it might be ignored if the scheduler determines the pod cannot go to a nominated node.Also, this field can be (over)written by the scheduler:If the scheduler finds a node to nominate via the preemption.If the scheduler decides where the pod is going, and move it to the binding cycle.Note that, in this case, nominatedNodeName is put only when the pod has to go through WaitOnPermit or PreBind extension points.Here is an example of a Pod status using the nominatedNodeName field:apiVersion: v1 kind: Pod metadata: name: nginx ... status: nominatedNodeName: kube-01 Pod topology spread constraintsYou can use topology spread constraints to control how Pods are spread across your cluster among failure-domains such as regions, zones, nodes, or among any other topology domains that you define. You might do this to improve performance, expected availability, or overall utilization.Read Pod topology spread constraints to learn more about how these work.OperatorsThe following are all the logical operators that you can use in the operator field for nodeAffinity and podAffinity mentioned above.OperatorBehaviorInThe label value is present in the supplied set of stringsNotInThe label value is not contained in the supplied set of stringsExistsA label with this key exists on the objectDoesNotExistNo label with this key exists on the objectThe following operators can only be used with nodeAffinity.OperatorBehaviorGtThe field value will be parsed as an integer, and that integer is less than the integer that results from parsing the value of a label named by this selectorLtThe field value will be parsed as an integer, and that integer is greater than the integer that results from parsing the value of a label named by this selectorNote:Gt and Lt operators will not work with non-integer values. If the given value doesn't parse as an integer, the Pod will fail to get scheduled. Also, Gt and Lt are not available for podAffinity.What's nextRead more about taints and tolerations.Read the design docs for node affinity and for inter-pod affinity/anti-affinity.Learn about how the topology manager takes part in node-level resource allocation decisions.Learn how to use nodeSelector.Learn how to use affinity and anti-affinity.


# Init Containers

Init ContainersThis page provides an overview of init containers: specialized containers that run before app containers in a Pod. Init containers can contain utilities or setup scripts not present in an app image.You can specify init containers in the Pod specification alongside the containers array (which describes app containers).In Kubernetes, a sidecar container is a container that starts before the main application container and continues to run. This document is about init containers: containers that run to completion during Pod initialization.Understanding init containersA Pod can have multiple containers running apps within it, but it can also have one or more init containers, which are run before the app containers are started.Init containers are exactly like regular containers, except:Init containers always run to completion.Each init container must complete successfully before the next one starts.If a Pod's init container fails, the kubelet repeatedly restarts that init container until it succeeds. However, if the Pod has a restartPolicy of Never, and an init container fails during startup of that Pod, Kubernetes treats the overall Pod as failed.To specify an init container for a Pod, add the initContainers field into the Pod specification, as an array of container items (similar to the app containers field and its contents). See Container in the API reference for more details.The status of the init containers is returned in .status.initContainerStatuses field as an array of the container statuses (similar to the .status.containerStatuses field).Differences from regular containersInit containers support all the fields and features of app containers, including resource limits, volumes, and security settings. However, the resource requests and limits for an init container are handled differently, as documented in Resource sharing within containers.Regular init containers (in other words: excluding sidecar containers) do not support the lifecycle, livenessProbe, readinessProbe, or startupProbe fields. Init containers must run to completion before the Pod can be ready; sidecar containers continue running during a Pod's lifetime, and do support some probes. See sidecar container for further details about sidecar containers.If you specify multiple init containers for a Pod, kubelet runs each init container sequentially. Each init container must succeed before the next can run. When all of the init containers have run to completion, kubelet initializes the application containers for the Pod and runs them as usual.Differences from sidecar containersInit containers run and complete their tasks before the main application container starts. Unlike sidecar containers, init containers are not continuously running alongside the main containers.Init containers run to completion sequentially, and the main container does not start until all the init containers have successfully completed.init containers do not support lifecycle, livenessProbe, readinessProbe, or startupProbe whereas sidecar containers support all these probes to control their lifecycle.Init containers share the same resources (CPU, memory, network) with the main application containers but do not interact directly with them. They can, however, use shared volumes for data exchange.Using init containersBecause init containers have separate images from app containers, they have some advantages for start-up related code:Init containers can contain utilities or custom code for setup that are not present in an app image. For example, there is no need to make an image FROM another image just to use a tool like sed, awk, python, or dig during setup.The application image builder and deployer roles can work independently without the need to jointly build a single app image.Init containers can run with a different view of the filesystem than app containers in the same Pod. Consequently, they can be given access to Secrets that app containers cannot access.Because init containers run to completion before any app containers start, init containers offer a mechanism to block or delay app container startup until a set of preconditions are met. Once preconditions are met, all of the app containers in a Pod can start in parallel.Init containers can securely run utilities or custom code that would otherwise make an app container image less secure. By keeping unnecessary tools separate you can limit the attack surface of your app container image.ExamplesHere are some ideas for how to use init containers:Wait for a Service to be created, using a shell one-line command like:for i in {1..100}; do sleep 1; if nslookup myservice; then exit 0; fi; done; exit 1 Register this Pod with a remote server from the downward API with a command like:curl -X POST http://$MANAGEMENT_SERVICE_HOST:$MANAGEMENT_SERVICE_PORT/register -d 'instance=$(<POD_NAME>)&ip=$(<POD_IP>)' Wait for some time before starting the app container with a command likesleep 60 Clone a Git repository into a VolumePlace values into a configuration file and run a template tool to dynamically generate a configuration file for the main app container. For example, place the POD_IP value in a configuration and generate the main app configuration file using Jinja.Init containers in useThis example defines a simple Pod that has two init containers. The first waits for myservice, and the second waits for mydb. Once both init containers complete, the Pod runs the app container from its spec section.apiVersion: v1 kind: Pod metadata: name: myapp-pod labels: app.kubernetes.io/name: MyApp spec: containers: - name: myapp-container image: busybox:1.28 command: ['sh', '-c', 'echo The app is running! && sleep 3600'] initContainers: - name: init-myservice image: busybox:1.28 command: ['sh', '-c', "until nslookup myservice.$(cat /var/run/secrets/kubernetes.io/serviceaccount/namespace).svc.cluster.local; do echo waiting for myservice; sleep 2; done"] - name: init-mydb image: busybox:1.28 command: ['sh', '-c', "until nslookup mydb.$(cat /var/run/secrets/kubernetes.io/serviceaccount/namespace).svc.cluster.local; do echo waiting for mydb; sleep 2; done"] You can start this Pod by running:kubectl apply -f myapp.yaml The output is similar to this:pod/myapp-pod created And check on its status with:kubectl get -f myapp.yaml The output is similar to this:NAME READY STATUS RESTARTS AGE myapp-pod 0/1 Init:0/2 0 6m or for more details:kubectl describe -f myapp.yaml The output is similar to this:Name: myapp-pod Namespace: default [...] Labels: app.kubernetes.io/name=MyApp Status: Pending [...] Init Containers: init-myservice: [...] State: Running [...] init-mydb: [...] State: Waiting Reason: PodInitializing Ready: False [...] Containers: myapp-container: [...] State: Waiting Reason: PodInitializing Ready: False [...] Events: FirstSeen LastSeen Count From SubObjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- 16s 16s 1 {default-scheduler } Normal Scheduled Successfully assigned myapp-pod to 172.17.4.201 16s 16s 1 {kubelet 172.17.4.201} spec.initContainers{init-myservice} Normal Pulling pulling image "busybox" 13s 13s 1 {kubelet 172.17.4.201} spec.initContainers{init-myservice} Normal Pulled Successfully pulled image "busybox" 13s 13s 1 {kubelet 172.17.4.201} spec.initContainers{init-myservice} Normal Created Created container init-myservice 13s 13s 1 {kubelet 172.17.4.201} spec.initContainers{init-myservice} Normal Started Started container init-myservice To see logs for the init containers in this Pod, run:kubectl logs myapp-pod -c init-myservice # Inspect the first init container kubectl logs myapp-pod -c init-mydb # Inspect the second init container At this point, those init containers will be waiting to discover Services named mydb and myservice.Here's a configuration you can use to make those Services appear:--- apiVersion: v1 kind: Service metadata: name: myservice spec: ports: - protocol: TCP port: 80 targetPort: 9376 --- apiVersion: v1 kind: Service metadata: name: mydb spec: ports: - protocol: TCP port: 80 targetPort: 9377 To create the mydb and myservice services:kubectl apply -f services.yaml The output is similar to this:service/myservice created service/mydb created You'll then see that those init containers complete, and that the myapp-pod Pod moves into the Running state:kubectl get -f myapp.yaml The output is similar to this:NAME READY STATUS RESTARTS AGE myapp-pod 1/1 Running 0 9m This simple example should provide some inspiration for you to create your own init containers. What's next contains a link to a more detailed example.Detailed behaviorDuring Pod startup, the kubelet delays running init containers until the networking and storage are ready. Then the kubelet runs the Pod's init containers in the order they appear in the Pod's spec.Each init container must exit successfully before the next container starts. If a container fails to start due to the runtime or exits with failure, it is retried according to the Pod restartPolicy. However, if the Pod restartPolicy is set to Always, the init containers use restartPolicy OnFailure.A Pod cannot be Ready until all init containers have succeeded. The ports on an init container are not aggregated under a Service. A Pod that is initializing is in the Pending state but should have a condition Initialized set to false.If the Pod restarts, or is restarted, all init containers must execute again.Changes to the init container spec are limited to the container image field. Directly altering the image field of an init container does not restart the Pod or trigger its recreation. If the Pod has yet to start, that change may have an effect on how the Pod boots up.For a pod template you can typically change any field for an init container; the impact of making that change depends on where the pod template is used.Because init containers can be restarted, retried, or re-executed, init container code should be idempotent. In particular, code that writes into any emptyDir volume should be prepared for the possibility that an output file already exists.Init containers have all of the fields of an app container. However, Kubernetes prohibits readinessProbe from being used because init containers cannot define readiness distinct from completion. This is enforced during validation.Use activeDeadlineSeconds on the Pod to prevent init containers from failing forever. The active deadline includes init containers. However it is recommended to use activeDeadlineSeconds only if teams deploy their application as a Job, because activeDeadlineSeconds has an effect even after initContainer finished. The Pod which is already running correctly would be killed by activeDeadlineSeconds if you set.The name of each app and init container in a Pod must be unique; a validation error is thrown for any container sharing a name with another.Resource sharing within containersGiven the order of execution for init, sidecar and app containers, the following rules for resource usage apply:The highest of any particular resource request or limit defined on all init containers is the effective init request/limit. If any resource has no resource limit specified this is considered as the highest limit.The Pod's effective request/limit for a resource is the higher of:the sum of all app containers request/limit for a resourcethe effective init request/limit for a resourceScheduling is done based on effective requests/limits, which means init containers can reserve resources for initialization that are not used during the life of the Pod.The QoS (quality of service) tier of the Pod's effective QoS tier is the QoS tier for init containers and app containers alike.Quota and limits are applied based on the effective Pod request and limit.Init containers and Linux cgroupsOn Linux, resource allocations for Pod level control groups (cgroups) are based on the effective Pod request and limit, the same as the scheduler.Pod restart reasonsA Pod can restart, causing re-execution of init containers, for the following reasons:The Pod infrastructure container is restarted. This is uncommon and would have to be done by someone with root access to nodes.All containers in a Pod are terminated while restartPolicy is set to Always, forcing a restart, and the init container completion record has been lost due to garbage collection.The Pod will not be restarted when the init container image is changed, or the init container completion record has been lost due to garbage collection. This applies for Kubernetes v1.20 and later. If you are using an earlier version of Kubernetes, consult the documentation for the version you are using.What's nextLearn more about the following:Creating a Pod that has an init container.Debug init containers.Overview of kubelet and kubectl.Types of probes: liveness, readiness, startup probe.Sidecar containers.


# Run a Stateless Application Using a Deployment

Run a Stateless Application Using a DeploymentThis page shows how to run an application using a Kubernetes Deployment object.ObjectivesCreate an nginx deployment.Use kubectl to list information about the deployment.Update the deployment.Before you beginYou need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. It is recommended to run this tutorial on a cluster with at least two nodes that are not acting as control plane hosts. If you do not already have a cluster, you can create one by using minikube or you can use one of these Kubernetes playgrounds:iximiuz LabsKillercodaKodeKloudPlay with KubernetesYour Kubernetes server must be at or later than version v1.9.To check the version, enter kubectl version.Creating and exploring an nginx deploymentYou can run an application by creating a Kubernetes Deployment object, and you can describe a Deployment in a YAML file. For example, this YAML file describes a Deployment that runs the nginx:1.14.2 Docker image:application/deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment spec: selector: matchLabels: app: nginx replicas: 2 # tells deployment to run 2 pods matching the template template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx:1.14.2 ports: - containerPort: 80 Create a Deployment based on the YAML file:kubectl apply -f https://k8s.io/examples/application/deployment.yaml Display information about the Deployment:kubectl describe deployment nginx-deployment The output is similar to this:Name: nginx-deployment Namespace: default CreationTimestamp: Tue, 30 Aug 2016 18:11:37 -0700 Labels: app=nginx Annotations: deployment.kubernetes.io/revision=1 Selector: app=nginx Replicas: 2 desired | 2 updated | 2 total | 2 available | 0 unavailable StrategyType: RollingUpdate MinReadySeconds: 0 RollingUpdateStrategy: 1 max unavailable, 1 max surge Pod Template: Labels: app=nginx Containers: nginx: Image: nginx:1.14.2 Port: 80/TCP Environment: <none> Mounts: <none> Volumes: <none> Conditions: Type Status Reason ---- ------ ------ Available True MinimumReplicasAvailable Progressing True NewReplicaSetAvailable OldReplicaSets: <none> NewReplicaSet: nginx-deployment-1771418926 (2/2 replicas created) No events. List the Pods created by the deployment:kubectl get pods -l app=nginx The output is similar to this:NAME READY STATUS RESTARTS AGE nginx-deployment-1771418926-7o5ns 1/1 Running 0 16h nginx-deployment-1771418926-r18az 1/1 Running 0 16h Display information about a Pod:kubectl describe pod <pod-name> where <pod-name> is the name of one of your Pods.Updating the deploymentYou can update the deployment by applying a new YAML file. This YAML file specifies that the deployment should be updated to use nginx 1.16.1.application/deployment-update.yaml apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment spec: selector: matchLabels: app: nginx replicas: 2 template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx:1.16.1 # Update the version of nginx from 1.14.2 to 1.16.1 ports: - containerPort: 80 Apply the new YAML file:kubectl apply -f https://k8s.io/examples/application/deployment-update.yaml Watch the deployment create pods with new names and delete the old pods:kubectl get pods -l app=nginx Scaling the application by increasing the replica countYou can increase the number of Pods in your Deployment by applying a new YAML file. This YAML file sets replicas to 4, which specifies that the Deployment should have four Pods:application/deployment-scale.yaml apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment spec: selector: matchLabels: app: nginx replicas: 4 # Update the replicas from 2 to 4 template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx:1.16.1 ports: - containerPort: 80 Apply the new YAML file:kubectl apply -f https://k8s.io/examples/application/deployment-scale.yaml Verify that the Deployment has four Pods:kubectl get pods -l app=nginx The output is similar to this:NAME READY STATUS RESTARTS AGE nginx-deployment-148880595-4zdqq 1/1 Running 0 25s nginx-deployment-148880595-6zgi1 1/1 Running 0 25s nginx-deployment-148880595-fxcez 1/1 Running 0 2m nginx-deployment-148880595-rwovn 1/1 Running 0 2m Deleting a deploymentDelete the deployment by name:kubectl delete deployment nginx-deployment ReplicationControllers -- the Old WayThe preferred way to create a replicated application is to use a Deployment, which in turn uses a ReplicaSet. Before the Deployment and ReplicaSet were added to Kubernetes, replicated applications were configured using a ReplicationController.What's nextLearn more about Deployment objects.


# Job with Pod-to-Pod Communication

Job with Pod-to-Pod CommunicationIn this example, you will run a Job in Indexed completion mode configured such that the pods created by the Job can communicate with each other using pod hostnames rather than pod IP addresses.Pods within a Job might need to communicate among themselves. The user workload running in each pod could query the Kubernetes API server to learn the IPs of the other Pods, but it's much simpler to rely on Kubernetes' built-in DNS resolution.Jobs in Indexed completion mode automatically set the pods' hostname to be in the format of ${jobName}-${completionIndex}. You can use this format to deterministically build pod hostnames and enable pod communication without needing to create a client connection to the Kubernetes control plane to obtain pod hostnames/IPs via API requests.This configuration is useful for use cases where pod networking is required but you don't want to depend on a network connection with the Kubernetes API server.Before you beginYou should already be familiar with the basic use of Job.You need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. It is recommended to run this tutorial on a cluster with at least two nodes that are not acting as control plane hosts. If you do not already have a cluster, you can create one by using minikube or you can use one of these Kubernetes playgrounds:iximiuz LabsKillercodaKodeKloudPlay with KubernetesYour Kubernetes server must be at or later than version v1.21.To check the version, enter kubectl version.Note:If you are using minikube or a similar tool, you may need to take extra steps to ensure you have DNS.Starting a Job with pod-to-pod communicationTo enable pod-to-pod communication using pod hostnames in a Job, you must do the following:Set up a headless Service with a valid label selector for the pods created by your Job. The headless service must be in the same namespace as the Job. One easy way to do this is to use the job-name: <your-job-name> selector, since the job-name label will be automatically added by Kubernetes. This configuration will trigger the DNS system to create records of the hostnames of the pods running your Job.Configure the headless service as subdomain service for the Job pods by including the following value in your Job template spec:subdomain: <headless-svc-name> ExampleBelow is a working example of a Job with pod-to-pod communication via pod hostnames enabled. The Job is completed only after all pods successfully ping each other using hostnames.Note:In the Bash script executed on each pod in the example below, the pod hostnames can be prefixed by the namespace as well if the pod needs to be reached from outside the namespace.apiVersion: v1 kind: Service metadata: name: headless-svc spec: clusterIP: None # clusterIP must be None to create a headless service selector: job-name: example-job # must match Job name --- apiVersion: batch/v1 kind: Job metadata: name: example-job spec: completions: 3 parallelism: 3 completionMode: Indexed template: spec: subdomain: headless-svc # has to match Service name restartPolicy: Never containers: - name: example-workload image: bash:latest command: - bash - -c - | for i in 0 1 2 do gotStatus="-1" wantStatus="0" while [ $gotStatus -ne $wantStatus ] do ping -c 1 example-job-${i}.headless-svc > /dev/null 2>&1 gotStatus=$? if [ $gotStatus -ne $wantStatus ]; then echo "Failed to ping pod example-job-${i}.headless-svc, retrying in 1 second..." sleep 1 fi done echo "Successfully pinged pod: example-job-${i}.headless-svc" done After applying the example above, reach each other over the network using: <pod-hostname>.<headless-service-name>. You should see output similar to the following:kubectl logs example-job-0-qws42 Failed to ping pod example-job-0.headless-svc, retrying in 1 second... Successfully pinged pod: example-job-0.headless-svc Successfully pinged pod: example-job-1.headless-svc Successfully pinged pod: example-job-2.headless-svc Note:Keep in mind that the <pod-hostname>.<headless-service-name> name format used in this example would not work with DNS policy set to None or Default. Refer to Pod's DNS Policy.