ClusterTrustBundle v1beta1ClusterTrustBundle is a cluster-scoped container for X.apiVersion: certificates.k8s.io/v1beta1import "k8s.io/api/certificates/v1beta1"ClusterTrustBundleClusterTrustBundle is a cluster-scoped container for X.509 trust anchors (root certificates).ClusterTrustBundle objects are considered to be readable by any authenticated user in the cluster, because they can be mounted by pods using the clusterTrustBundle projection. All service accounts have read access to ClusterTrustBundles by default. Users who only have namespace-level access to a cluster can read ClusterTrustBundles by impersonating a serviceaccount that they have access to.It can be optionally associated with a particular assigner, in which case it contains one valid set of trust anchors for that signer. Signers may have multiple associated ClusterTrustBundles; each is an independent set of trust anchors for that signer. Admission control is used to enforce that only users with permissions on the signer can create or modify the corresponding bundle.apiVersion: certificates.k8s.io/v1beta1kind: ClusterTrustBundlemetadata (ObjectMeta)metadata contains the object metadata.spec (ClusterTrustBundleSpec), requiredspec contains the signer (if any) and trust anchors.ClusterTrustBundleSpecClusterTrustBundleSpec contains the signer and trust anchors.trustBundle (string), requiredtrustBundle contains the individual X.509 trust anchors for this bundle, as PEM bundle of PEM-wrapped, DER-formatted X.509 certificates.The data must consist only of PEM certificate blocks that parse as valid X.509 certificates. Each certificate must include a basic constraints extension with the CA bit set. The API server will reject objects that contain duplicate certificates, or that use PEM block headers.Users of ClusterTrustBundles, including Kubelet, are free to reorder and deduplicate certificate blocks in this file according to their own logic, as well as to drop PEM block headers and inter-block data.signerName (string)signerName indicates the associated signer, if any.In order to create or update a ClusterTrustBundle that sets signerName, you must have the following cluster-scoped permission: group=certificates.k8s.io resource=signers resourceName=<the signer name> verb=attest.If signerName is not empty, then the ClusterTrustBundle object must be named with the signer name as a prefix (translating slashes to colons). For example, for the signer name example.com/foo, valid ClusterTrustBundle object names include example.com:foo:abc and example.com:foo:v1.If signerName is empty, then the ClusterTrustBundle object's name must not have such a prefix.List/watch requests for ClusterTrustBundles can filter on this field using a spec.signerName=NAME field selector.ClusterTrustBundleListClusterTrustBundleList is a collection of ClusterTrustBundle objectsapiVersion: certificates.k8s.io/v1beta1kind: ClusterTrustBundleListmetadata (ListMeta)metadata contains the list metadata.items ([]ClusterTrustBundle), requireditems is a collection of ClusterTrustBundle objectsOperationsget read the specified ClusterTrustBundleHTTP RequestGET /apis/certificates.k8s.io/v1beta1/clustertrustbundles/{name}Parametersname (in path): string, requiredname of the ClusterTrustBundlepretty (in query): stringprettyResponse200 (ClusterTrustBundle): OK401: Unauthorizedlist list or watch objects of kind ClusterTrustBundleHTTP RequestGET /apis/certificates.k8s.io/v1beta1/clustertrustbundlesParametersallowWatchBookmarks (in query): booleanallowWatchBookmarkscontinue (in query): stringcontinuefieldSelector (in query): stringfieldSelectorlabelSelector (in query): stringlabelSelectorlimit (in query): integerlimitpretty (in query): stringprettyresourceVersion (in query): stringresourceVersionresourceVersionMatch (in query): stringresourceVersionMatchsendInitialEvents (in query): booleansendInitialEventstimeoutSeconds (in query): integertimeoutSecondswatch (in query): booleanwatchResponse200 (ClusterTrustBundleList): OK401: Unauthorizedcreate create a ClusterTrustBundleHTTP RequestPOST /apis/certificates.k8s.io/v1beta1/clustertrustbundlesParametersbody: ClusterTrustBundle, requireddryRun (in query): stringdryRunfieldManager (in query): stringfieldManagerfieldValidation (in query): stringfieldValidationpretty (in query): stringprettyResponse200 (ClusterTrustBundle): OK201 (ClusterTrustBundle): Created202 (ClusterTrustBundle): Accepted401: Unauthorizedupdate replace the specified ClusterTrustBundleHTTP RequestPUT /apis/certificates.k8s.io/v1beta1/clustertrustbundles/{name}Parametersname (in path): string, requiredname of the ClusterTrustBundlebody: ClusterTrustBundle, requireddryRun (in query): stringdryRunfieldManager (in query): stringfieldManagerfieldValidation (in query): stringfieldValidationpretty (in query): stringprettyResponse200 (ClusterTrustBundle): OK201 (ClusterTrustBundle): Created401: Unauthorizedpatch partially update the specified ClusterTrustBundleHTTP RequestPATCH /apis/certificates.k8s.io/v1beta1/clustertrustbundles/{name}Parametersname (in path): string, requiredname of the ClusterTrustBundlebody: Patch, requireddryRun (in query): stringdryRunfieldManager (in query): stringfieldManagerfieldValidation (in query): stringfieldValidationforce (in query): booleanforcepretty (in query): stringprettyResponse200 (ClusterTrustBundle): OK201 (ClusterTrustBundle): Created401: Unauthorizeddelete delete a ClusterTrustBundleHTTP RequestDELETE /apis/certificates.k8s.io/v1beta1/clustertrustbundles/{name}Parametersname (in path): string, requiredname of the ClusterTrustBundlebody: DeleteOptionsdryRun (in query): stringdryRungracePeriodSeconds (in query): integergracePeriodSecondsignoreStoreReadErrorWithClusterBreakingPotential (in query): booleanignoreStoreReadErrorWithClusterBreakingPotentialpretty (in query): stringprettypropagationPolicy (in query): stringpropagationPolicyResponse200 (Status): OK202 (Status): Accepted401: Unauthorizeddeletecollection delete collection of ClusterTrustBundleHTTP RequestDELETE /apis/certificates.k8s.io/v1beta1/clustertrustbundlesParametersbody: DeleteOptionscontinue (in query): stringcontinuedryRun (in query): stringdryRunfieldSelector (in query): stringfieldSelectorgracePeriodSeconds (in query): integergracePeriodSecondsignoreStoreReadErrorWithClusterBreakingPotential (in query): booleanignoreStoreReadErrorWithClusterBreakingPotentiallabelSelector (in query): stringlabelSelectorlimit (in query): integerlimitpretty (in query): stringprettypropagationPolicy (in query): stringpropagationPolicyresourceVersion (in query): stringresourceVersionresourceVersionMatch (in query): stringresourceVersionMatchsendInitialEvents (in query): booleansendInitialEventstimeoutSeconds (in query): integertimeoutSecondsResponse200 (Status): OK401: UnauthorizedThis page is automatically generated.If you plan to report an issue with this page, mention that the page is auto-generated in your issue description. The fix may need to happen elsewhere in the Kubernetes project.

ClusterTrustBundle v1beta1


kubectl certificateSynopsisModify certificate resources.kubectl certificate SUBCOMMAND Options-h, --helphelp for certificateParent Options Inherited--as stringUsername to impersonate for the operation. User could be a regular user or a service account in a namespace.--as-group stringsGroup to impersonate for the operation, this flag can be repeated to specify multiple groups.--as-uid stringUID to impersonate for the operation.--cache-dir string Default: "$HOME/.kube/cache"Default cache directory--certificate-authority stringPath to a cert file for the certificate authority--client-certificate stringPath to a client certificate file for TLS--client-key stringPath to a client key file for TLS--cluster stringThe name of the kubeconfig cluster to use--context stringThe name of the kubeconfig context to use--disable-compressionIf true, opt-out of response compression for all requests to the server--insecure-skip-tls-verifyIf true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure--kubeconfig stringPath to the kubeconfig file to use for CLI requests.--kuberc stringPath to the kuberc file to use for preferences. This can be disabled by exporting KUBECTL_KUBERC=false feature gate or turning off the feature KUBERC=off.--match-server-versionRequire server version to match client version-n, --namespace stringIf present, the namespace scope for this CLI request--password stringPassword for basic authentication to the API server--profile string Default: "none"Name of profile to capture. One of (none|cpu|heap|goroutine|threadcreate|block|mutex)--profile-output string Default: "profile.pprof"Name of the file to write the profile to--request-timeout string Default: "0"The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests.-s, --server stringThe address and port of the Kubernetes API server--storage-driver-buffer-duration duration Default: 1m0sWrites in the storage driver will be buffered for this duration, and committed to the non memory backends as a single transaction--storage-driver-db string Default: "cadvisor"database name--storage-driver-host string Default: "localhost:8086"database host:port--storage-driver-password string Default: "root"database password--storage-driver-secureuse secure connection with database--storage-driver-table string Default: "stats"table name--storage-driver-user string Default: "root"database username--tls-server-name stringServer name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used--token stringBearer token for authentication to the API server--user stringThe name of the kubeconfig user to use--username stringUsername for basic authentication to the API server--version version[=true]--version, --version=raw prints version information and quits; --version=vX.Y.Z... sets the reported version--warnings-as-errorsTreat warnings received from the server as errors and exit with a non-zero exit codeSee Alsokubectl - kubectl controls the Kubernetes cluster managerkubectl certificate approve - Approve a certificate signing requestkubectl certificate deny - Deny a certificate signing requestThis page is automatically generated.If you plan to report an issue with this page, mention that the page is auto-generated in your issue description. The fix may need to happen elsewhere in the Kubernetes project.

kubectl certificate


kubectl exposeSynopsisExpose a resource as a new Kubernetes service.Looks up a deployment, service, replica set, replication controller or pod by name and uses the selector for that resource as the selector for a new service on the specified port. A deployment or replica set will be exposed as a service only if its selector is convertible to a selector that service supports, i.e. when the selector contains only the matchLabels component. Note that if no port is specified via --port and the exposed resource has multiple ports, all will be re-used by the new service. Also if no labels are specified, the new service will re-use the labels from the resource it exposes.Possible resources include (case insensitive):pod (po), service (svc), replicationcontroller (rc), deployment (deploy), replicaset (rs)kubectl expose (-f FILENAME | TYPE NAME) [--port=port] [--protocol=TCP|UDP|SCTP] [--target-port=number-or-name] [--name=name] [--external-ip=external-ip-of-service] [--type=type] Examples # Create a service for a replicated nginx, which serves on port 80 and connects to the containers on port 8000 kubectl expose rc nginx --port=80 --target-port=8000 # Create a service for a replication controller identified by type and name specified in "nginx-controller.yaml", which serves on port 80 and connects to the containers on port 8000 kubectl expose -f nginx-controller.yaml --port=80 --target-port=8000 # Create a service for a pod valid-pod, which serves on port 444 with the name "frontend" kubectl expose pod valid-pod --port=444 --name=frontend # Create a second service based on the above service, exposing the container port 8443 as port 443 with the name "nginx-https" kubectl expose service nginx --port=443 --target-port=8443 --name=nginx-https # Create a service for a replicated streaming application on port 4100 balancing UDP traffic and named 'video-stream'. kubectl expose rc streamer --port=4100 --protocol=UDP --name=video-stream # Create a service for a replicated nginx using replica set, which serves on port 80 and connects to the containers on port 8000 kubectl expose rs nginx --port=80 --target-port=8000 # Create a service for an nginx deployment, which serves on port 80 and connects to the containers on port 8000 kubectl expose deployment nginx --port=80 --target-port=8000 Options--allow-missing-template-keys Default: trueIf true, ignore any errors in templates when a field or map key is missing in the template. Only applies to golang and jsonpath output formats.--cluster-ip stringClusterIP to be assigned to the service. Leave empty to auto-allocate, or set to 'None' to create a headless service.--dry-run string[="unchanged"] Default: "none"Must be "none", "server", or "client". If client strategy, only print the object that would be sent, without sending it. If server strategy, submit server-side request without persisting the resource.--external-ip stringAdditional external IP address (not managed by Kubernetes) to accept for the service. If this IP is routed to a node, the service can be accessed by this IP in addition to its generated service IP.--field-manager string Default: "kubectl-expose"Name of the manager used to track field ownership.-f, --filename stringsFilename, directory, or URL to files identifying the resource to expose a service-h, --helphelp for expose-k, --kustomize stringProcess the kustomization directory. This flag can't be used together with -f or -R.-l, --labels stringLabels to apply to the service created by this call.--load-balancer-ip stringIP to assign to the LoadBalancer. If empty, an ephemeral IP will be created and used (cloud-provider specific).--name stringThe name for the newly created object.-o, --output stringOutput format. One of: (json, yaml, name, go-template, go-template-file, template, templatefile, jsonpath, jsonpath-as-json, jsonpath-file).--override-type string Default: "merge"The method used to override the generated object: json, merge, or strategic.--overrides stringAn inline JSON override for the generated object. If this is non-empty, it is used to override the generated object. Requires that the object supply a valid apiVersion field.--port stringThe port that the service should serve on. Copied from the resource being exposed, if unspecified--protocol stringThe network protocol for the service to be created. Default is 'TCP'.-R, --recursiveProcess the directory used in -f, --filename recursively. Useful when you want to manage related manifests organized within the same directory.--save-configIf true, the configuration of current object will be saved in its annotation. Otherwise, the annotation will be unchanged. This flag is useful when you want to perform kubectl apply on this object in the future.--selector stringA label selector to use for this service. Only equality-based selector requirements are supported. If empty (the default) infer the selector from the replication controller or replica set.)--session-affinity stringIf non-empty, set the session affinity for the service to this; legal values: 'None', 'ClientIP'--show-managed-fieldsIf true, keep the managedFields when printing objects in JSON or YAML format.--target-port stringName or number for the port on the container that the service should direct traffic to. Optional.--template stringTemplate string or path to template file to use when -o=go-template, -o=go-template-file. The template format is golang templates [http://golang.org/pkg/text/template/#pkg-overview].--type stringType for this service: ClusterIP, NodePort, LoadBalancer, or ExternalName. Default is 'ClusterIP'.Parent Options Inherited--as stringUsername to impersonate for the operation. User could be a regular user or a service account in a namespace.--as-group stringsGroup to impersonate for the operation, this flag can be repeated to specify multiple groups.--as-uid stringUID to impersonate for the operation.--cache-dir string Default: "$HOME/.kube/cache"Default cache directory--certificate-authority stringPath to a cert file for the certificate authority--client-certificate stringPath to a client certificate file for TLS--client-key stringPath to a client key file for TLS--cluster stringThe name of the kubeconfig cluster to use--context stringThe name of the kubeconfig context to use--disable-compressionIf true, opt-out of response compression for all requests to the server--insecure-skip-tls-verifyIf true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure--kubeconfig stringPath to the kubeconfig file to use for CLI requests.--kuberc stringPath to the kuberc file to use for preferences. This can be disabled by exporting KUBECTL_KUBERC=false feature gate or turning off the feature KUBERC=off.--match-server-versionRequire server version to match client version-n, --namespace stringIf present, the namespace scope for this CLI request--password stringPassword for basic authentication to the API server--profile string Default: "none"Name of profile to capture. One of (none|cpu|heap|goroutine|threadcreate|block|mutex)--profile-output string Default: "profile.pprof"Name of the file to write the profile to--request-timeout string Default: "0"The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests.-s, --server stringThe address and port of the Kubernetes API server--storage-driver-buffer-duration duration Default: 1m0sWrites in the storage driver will be buffered for this duration, and committed to the non memory backends as a single transaction--storage-driver-db string Default: "cadvisor"database name--storage-driver-host string Default: "localhost:8086"database host:port--storage-driver-password string Default: "root"database password--storage-driver-secureuse secure connection with database--storage-driver-table string Default: "stats"table name--storage-driver-user string Default: "root"database username--tls-server-name stringServer name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used--token stringBearer token for authentication to the API server--user stringThe name of the kubeconfig user to use--username stringUsername for basic authentication to the API server--version version[=true]--version, --version=raw prints version information and quits; --version=vX.Y.Z... sets the reported version--warnings-as-errorsTreat warnings received from the server as errors and exit with a non-zero exit codeSee Alsokubectl - kubectl controls the Kubernetes cluster managerThis page is automatically generated.If you plan to report an issue with this page, mention that the page is auto-generated in your issue description. The fix may need to happen elsewhere in the Kubernetes project.

kubectl expose


Gateway APIGateway API is a family of API kinds that provide dynamic infrastructure provisioning and advanced traffic routing.Make network services available by using an extensible, role-oriented, protocol-aware configuration mechanism. Gateway API is an add-on containing API kinds that provide dynamic infrastructure provisioning and advanced traffic routing.Design principlesThe following principles shaped the design and architecture of Gateway API:Role-oriented: Gateway API kinds are modeled after organizational roles that are responsible for managing Kubernetes service networking:Infrastructure Provider: Manages infrastructure that allows multiple isolated clusters to serve multiple tenants, e.g. a cloud provider.Cluster Operator: Manages clusters and is typically concerned with policies, network access, application permissions, etc.Application Developer: Manages an application running in a cluster and is typically concerned with application-level configuration and Service composition.Portable: Gateway API specifications are defined as custom resources and are supported by many implementations.Expressive: Gateway API kinds support functionality for common traffic routing use cases such as header-based matching, traffic weighting, and others that were only possible in Ingress by using custom annotations.Extensible: Gateway allows for custom resources to be linked at various layers of the API. This makes granular customization possible at the appropriate places within the API structure.Resource modelGateway API has four stable API kinds:GatewayClass: Defines a set of gateways with common configuration and managed by a controller that implements the class.Gateway: Defines an instance of traffic handling infrastructure, such as cloud load balancer.HTTPRoute: Defines HTTP-specific rules for mapping traffic from a Gateway listener to a representation of backend network endpoints. These endpoints are often represented as a Service.GRPCRoute: Defines gRPC-specific rules for mapping traffic from a Gateway listener to a representation of backend network endpoints. These endpoints are often represented as a Service.Gateway API is organized into different API kinds that have interdependent relationships to support the role-oriented nature of organizations. A Gateway object is associated with exactly one GatewayClass; the GatewayClass describes the gateway controller responsible for managing Gateways of this class. One or more route kinds such as HTTPRoute, are then associated to Gateways. A Gateway can filter the routes that may be attached to its listeners, forming a bidirectional trust model with routes.The following figure illustrates the relationships of the three stable Gateway API kinds:GatewayClassGateways can be implemented by different controllers, often with different configurations. A Gateway must reference a GatewayClass that contains the name of the controller that implements the class.A minimal GatewayClass example:apiVersion: gateway.networking.k8s.io/v1 kind: GatewayClass metadata: name: example-class spec: controllerName: example.com/gateway-controller In this example, a controller that has implemented Gateway API is configured to manage GatewayClasses with the controller name example.com/gateway-controller. Gateways of this class will be managed by the implementation's controller.See the GatewayClass reference for a full definition of this API kind.GatewayA Gateway describes an instance of traffic handling infrastructure. It defines a network endpoint that can be used for processing traffic, i.e. filtering, balancing, splitting, etc. for backends such as a Service. For example, a Gateway may represent a cloud load balancer or an in-cluster proxy server that is configured to accept HTTP traffic.A minimal Gateway resource example:apiVersion: gateway.networking.k8s.io/v1 kind: Gateway metadata: name: example-gateway spec: gatewayClassName: example-class listeners: - name: http protocol: HTTP port: 80 In this example, an instance of traffic handling infrastructure is programmed to listen for HTTP traffic on port 80. Since the addresses field is unspecified, an address or hostname is assigned to the Gateway by the implementation's controller. This address is used as a network endpoint for processing traffic of backend network endpoints defined in routes.See the Gateway reference for a full definition of this API kind.HTTPRouteThe HTTPRoute kind specifies routing behavior of HTTP requests from a Gateway listener to backend network endpoints. For a Service backend, an implementation may represent the backend network endpoint as a Service IP or the backing EndpointSlices of the Service. An HTTPRoute represents configuration that is applied to the underlying Gateway implementation. For example, defining a new HTTPRoute may result in configuring additional traffic routes in a cloud load balancer or in-cluster proxy server.A minimal HTTPRoute example:apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute metadata: name: example-httproute spec: parentRefs: - name: example-gateway hostnames: - "www.example.com" rules: - matches: - path: type: PathPrefix value: /login backendRefs: - name: example-svc port: 8080 In this example, HTTP traffic from Gateway example-gateway with the Host: header set to www.example.com and the request path specified as /login will be routed to Service example-svc on port 8080.See the HTTPRoute reference for a full definition of this API kind.GRPCRouteThe GRPCRoute kind specifies routing behavior of gRPC requests from a Gateway listener to backend network endpoints. For a Service backend, an implementation may represent the backend network endpoint as a Service IP or the backing EndpointSlices of the Service. A GRPCRoute represents configuration that is applied to the underlying Gateway implementation. For example, defining a new GRPCRoute may result in configuring additional traffic routes in a cloud load balancer or in-cluster proxy server.Gateways supporting GRPCRoute are required to support HTTP/2 without an initial upgrade from HTTP/1, so gRPC traffic is guaranteed to flow properly.A minimal GRPCRoute example:apiVersion: gateway.networking.k8s.io/v1 kind: GRPCRoute metadata: name: example-grpcroute spec: parentRefs: - name: example-gateway hostnames: - "svc.example.com" rules: - backendRefs: - name: example-svc port: 50051 In this example, gRPC traffic from Gateway example-gateway with the host set to svc.example.com will be directed to the service example-svc on port 50051 from the same namespace.GRPCRoute allows matching specific gRPC services, as per the following example:apiVersion: gateway.networking.k8s.io/v1 kind: GRPCRoute metadata: name: example-grpcroute spec: parentRefs: - name: example-gateway hostnames: - "svc.example.com" rules: - matches: - method: service: com.example method: Login backendRefs: - name: foo-svc port: 50051 In this case, the GRPCRoute will match any traffic for svc.example.com and apply its routing rules to forward the traffic to the correct backend. Since there is only one match specified,only requests for the com.example.User.Login method to svc.example.com will be forwarded. RPCs of any other method` will not be matched by this Route.See the GRPCRoute reference for a full definition of this API kind.Request flowHere is a simple example of HTTP traffic being routed to a Service by using a Gateway and an HTTPRoute:In this example, the request flow for a Gateway implemented as a reverse proxy is:The client starts to prepare an HTTP request for the URL http://www.example.comThe client's DNS resolver queries for the destination name and learns a mapping to one or more IP addresses associated with the Gateway.The client sends a request to the Gateway IP address; the reverse proxy receives the HTTP request and uses the Host: header to match a configuration that was derived from the Gateway and attached HTTPRoute.Optionally, the reverse proxy can perform request header and/or path matching based on match rules of the HTTPRoute.Optionally, the reverse proxy can modify the request; for example, to add or remove headers, based on filter rules of the HTTPRoute.Lastly, the reverse proxy forwards the request to one or more backends.ConformanceGateway API covers a broad set of features and is widely implemented. This combination requires clear conformance definitions and tests to ensure that the API provides a consistent experience wherever it is used.See the conformance documentation to understand details such as release channels, support levels, and running conformance tests.Migrating from IngressGateway API is the successor to the Ingress API. However, it does not include the Ingress kind. As a result, a one-time conversion from your existing Ingress resources to Gateway API resources is necessary.Refer to the ingress migration guide for details on migrating Ingress resources to Gateway API resources.What's nextInstead of Gateway API resources being natively implemented by Kubernetes, the specifications are defined as Custom Resources supported by a wide range of implementations. Install the Gateway API CRDs or follow the installation instructions of your selected implementation. After installing an implementation, use the Getting Started guide to help you quickly start working with Gateway API.Note:Make sure to review the documentation of your selected implementation to understand any caveats.Refer to the API specification for additional details of all Gateway API kinds.

Gateway API


Kubernetes API Server Bypass RisksSecurity architecture information relating to the API server and other componentsThe Kubernetes API server is the main point of entry to a cluster for external parties (users and services) interacting with it.As part of this role, the API server has several key built-in security controls, such as audit logging and admission controllers. However, there are ways to modify the configuration or content of the cluster that bypass these controls.This page describes the ways in which the security controls built into the Kubernetes API server can be bypassed, so that cluster operators and security architects can ensure that these bypasses are appropriately restricted.Static PodsThe kubelet on each node loads and directly manages any manifests that are stored in a named directory or fetched from a specific URL as static Pods in your cluster. The API server doesn't manage these static Pods. An attacker with write access to this location could modify the configuration of static pods loaded from that source, or could introduce new static Pods.Static Pods are restricted from accessing other objects in the Kubernetes API. For example, you can't configure a static Pod to mount a Secret from the cluster. However, these Pods can take other security sensitive actions, such as using hostPath mounts from the underlying node.By default, the kubelet creates a mirror pod so that the static Pods are visible in the Kubernetes API. However, if the attacker uses an invalid namespace name when creating the Pod, it will not be visible in the Kubernetes API and can only be discovered by tooling that has access to the affected host(s).If a static Pod fails admission control, the kubelet won't register the Pod with the API server. However, the Pod still runs on the node. For more information, refer to kubeadm issue #1541.MitigationsOnly enable the kubelet static Pod manifest functionality if required by the node.If a node uses the static Pod functionality, restrict filesystem access to the static Pod manifest directory or URL to users who need the access.Restrict access to kubelet configuration parameters and files to prevent an attacker setting a static Pod path or URL.Regularly audit and centrally report all access to directories or web storage locations that host static Pod manifests and kubelet configuration files.The kubelet APIThe kubelet provides an HTTP API that is typically exposed on TCP port 10250 on cluster worker nodes. The API might also be exposed on control plane nodes depending on the Kubernetes distribution in use. Direct access to the API allows for disclosure of information about the pods running on a node, the logs from those pods, and execution of commands in every container running on the node.When Kubernetes cluster users have RBAC access to Node object sub-resources, that access serves as authorization to interact with the kubelet API. The exact access depends on which sub-resource access has been granted, as detailed in kubelet authorization.Direct access to the kubelet API is not subject to admission control and is not logged by Kubernetes audit logging. An attacker with direct access to this API may be able to bypass controls that detect or prevent certain actions.The kubelet API can be configured to authenticate requests in a number of ways. By default, the kubelet configuration allows anonymous access. Most Kubernetes providers change the default to use webhook and certificate authentication. This lets the control plane ensure that the caller is authorized to access the nodes API resource or sub-resources. The default anonymous access doesn't make this assertion with the control plane.MitigationsRestrict access to sub-resources of the nodes API object using mechanisms such as RBAC. Only grant this access when required, such as by monitoring services.Restrict access to the kubelet port. Only allow specified and trusted IP address ranges to access the port.Ensure that kubelet authentication. is set to webhook or certificate mode.Ensure that the unauthenticated "read-only" Kubelet port is not enabled on the cluster.The etcd APIKubernetes clusters use etcd as a datastore. The etcd service listens on TCP port 2379. The only clients that need access are the Kubernetes API server and any backup tooling that you use. Direct access to this API allows for disclosure or modification of any data held in the cluster.Access to the etcd API is typically managed by client certificate authentication. Any certificate issued by a certificate authority that etcd trusts allows full access to the data stored inside etcd.Direct access to etcd is not subject to Kubernetes admission control and is not logged by Kubernetes audit logging. An attacker who has read access to the API server's etcd client certificate private key (or can create a new trusted client certificate) can gain cluster admin rights by accessing cluster secrets or modifying access rules. Even without elevating their Kubernetes RBAC privileges, an attacker who can modify etcd can retrieve any API object or create new workloads inside the cluster.Many Kubernetes providers configure etcd to use mutual TLS (both client and server verify each other's certificate for authentication). There is no widely accepted implementation of authorization for the etcd API, although the feature exists. Since there is no authorization model, any certificate with client access to etcd can be used to gain full access to etcd. Typically, etcd client certificates that are only used for health checking can also grant full read and write access.MitigationsEnsure that the certificate authority trusted by etcd is used only for the purposes of authentication to that service.Control access to the private key for the etcd server certificate, and to the API server's client certificate and key.Consider restricting access to the etcd port at a network level, to only allow access from specified and trusted IP address ranges.Container runtime socketOn each node in a Kubernetes cluster, access to interact with containers is controlled by the container runtime (or runtimes, if you have configured more than one). Typically, the container runtime exposes a Unix socket that the kubelet can access. An attacker with access to this socket can launch new containers or interact with running containers.At the cluster level, the impact of this access depends on whether the containers that run on the compromised node have access to Secrets or other confidential data that an attacker could use to escalate privileges to other worker nodes or to control plane components.MitigationsEnsure that you tightly control filesystem access to container runtime sockets. When possible, restrict this access to the root user.Isolate the kubelet from other components running on the node, using mechanisms such as Linux kernel namespaces.Ensure that you restrict or forbid the use of hostPath mounts that include the container runtime socket, either directly or by mounting a parent directory. Also hostPath mounts must be set as read-only to mitigate risks of attackers bypassing directory restrictions.Restrict user access to nodes, and especially restrict superuser access to nodes.

Kubernetes API Server Bypass Risks


Using a Service to Expose Your AppObjectivesLearn about a Service in Kubernetes.Understand how labels and selectors relate to a Service.Expose an application outside a Kubernetes cluster.Overview of Kubernetes ServicesKubernetes Pods are mortal. Pods have a lifecycle. When a worker node dies, the Pods running on the Node are also lost. A Replicaset might then dynamically drive the cluster back to the desired state via the creation of new Pods to keep your application running. As another example, consider an image-processing backend with 3 replicas. Those replicas are exchangeable; the front-end system should not care about backend replicas or even if a Pod is lost and recreated. That said, each Pod in a Kubernetes cluster has a unique IP address, even Pods on the same Node, so there needs to be a way of automatically reconciling changes among Pods so that your applications continue to function.A Kubernetes Service is an abstraction layer which defines a logical set of Pods and enables external traffic exposure, load balancing and service discovery for those Pods.A Service in Kubernetes is an abstraction which defines a logical set of Pods and a policy by which to access them. Services enable a loose coupling between dependent Pods. A Service is defined using YAML or JSON, like all Kubernetes object manifests. The set of Pods targeted by a Service is usually determined by a label selector (see below for why you might want a Service without including a selector in the spec).Although each Pod has a unique IP address, those IPs are not exposed outside the cluster without a Service. Services allow your applications to receive traffic. Services can be exposed in different ways by specifying a type in the spec of the Service:ClusterIP (default) - Exposes the Service on an internal IP in the cluster. This type makes the Service only reachable from within the cluster.NodePort - Exposes the Service on the same port of each selected Node in the cluster using NAT. Makes a Service accessible from outside the cluster using NodeIP:NodePort. Superset of ClusterIP.LoadBalancer - Creates an external load balancer in the current cloud (if supported) and assigns a fixed, external IP to the Service. Superset of NodePort.ExternalName - Maps the Service to the contents of the externalName field (e.g. foo.bar.example.com), by returning a CNAME record with its value. No proxying of any kind is set up. This type requires v1.7 or higher of kube-dns, or CoreDNS version 0.0.8 or higher.More information about the different types of Services can be found in the Using Source IP tutorial. Also see Connecting Applications with Services.Additionally, note that there are some use cases with Services that involve not defining a selector in the spec. A Service created without selector will also not create the corresponding Endpoints object. This allows users to manually map a Service to specific endpoints. Another possibility why there may be no selector is you are strictly using type: ExternalName.Services and LabelsA Service routes traffic across a set of Pods. Services are the abstraction that allows pods to die and replicate in Kubernetes without impacting your application. Discovery and routing among dependent Pods (such as the frontend and backend components in an application) are handled by Kubernetes Services.Services match a set of Pods using labels and selectors, a grouping primitive that allows logical operation on objects in Kubernetes. Labels are key/value pairs attached to objects and can be used in any number of ways:Designate objects for development, test, and productionEmbed version tagsClassify an object using tagsLabels can be attached to objects at creation time or later on. They can be modified at any time. Let's expose our application now using a Service and apply some labels.Step 1: Creating a new ServiceLet’s verify that our application is running. We’ll use the kubectl get command and look for existing Pods:kubectl get pods If no Pods are running then it means the objects from the previous tutorials were cleaned up. In this case, go back and recreate the deployment from the Using kubectl to create a Deployment tutorial. Please wait a couple of seconds and list the Pods again. You can continue once you see the one Pod running.

Using a Service to Expose Your App


Assign Pod-level CPU and memory resourcesFEATURE STATE: Kubernetes v1.34 [beta] (enabled by default: true)This page shows how to specify CPU and memory resources for a Pod at pod-level in addition to container-level resource specifications. A Kubernetes node allocates resources to a pod based on the pod's resource requests. These requests can be defined at the pod level or individually for containers within the pod. When both are present, the pod-level requests take precedence.Similarly, a pod's resource usage is restricted by limits, which can also be set at the pod-level or individually for containers within the pod. Again, pod-level limits are prioritized when both are present. This allows for flexible resource management, enabling you to control resource allocation at both the pod and container levels.In order to specify the resources at pod-level, it is required to enable PodLevelResources feature gate.For Pod Level Resources:Priority: When both pod-level and container-level resources are specified, pod-level resources take precedence.QoS: Pod-level resources take precedence in influencing the QoS class of the pod.OOM Score: The OOM score adjustment calculation considers both pod-level and container-level resources.Compatibility: Pod-level resources are designed to be compatible with existing features.Before you beginYou need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. It is recommended to run this tutorial on a cluster with at least two nodes that are not acting as control plane hosts. If you do not already have a cluster, you can create one by using minikube or you can use one of these Kubernetes playgrounds:iximiuz LabsKillercodaKodeKloudPlay with KubernetesYour Kubernetes server must be at or later than version 1.34.To check the version, enter kubectl version.The PodLevelResources feature gate must be enabled for your control plane and for all nodes in your cluster.LimitationsFor Kubernetes 1.34, resizing pod-level resources has the following limitations:Resource Types: Only CPU, memory and hugepages resources can be specified at pod-level.Operating System: Pod-level resources are not supported for Windows pods.Resource Managers: The Topology Manager, Memory Manager and CPU Manager do not align pods and containers based on pod-level resources as these resource managers don't currently support pod-level resources.In-Place Resize: In-place resize of pod-level resources is not supported. Modifying the pod-level resource limits or requests on a pod result in a field.Forbidden error. The error message explicitly states, "pods with pod-level resources cannot be resized."Create a namespaceCreate a namespace so that the resources you create in this exercise are isolated from the rest of your cluster.kubectl create namespace pod-resources-example Create a pod with memory requests and limits at pod-levelTo specify memory requests for a Pod at pod-level, include the resources.requests.memory field in the Pod spec manifest. To specify a memory limit, include resources.limits.memory.In this exercise, you create a Pod that has one Container. The Pod has a memory request of 100 MiB and a memory limit of 200 MiB. Here's the configuration file for the Pod:pods/resource/pod-level-memory-request-limit.yaml apiVersion: v1 kind: Pod metadata: name: memory-demo namespace: pod-resources-example spec: resources: requests: memory: "100Mi" limits: memory: "200Mi" containers: - name: memory-demo-ctr image: nginx command: ["stress"] args: ["--vm", "1", "--vm-bytes", "150M", "--vm-hang", "1"] The args section in the manifest provides arguments for the container when it starts. The "--vm-bytes", "150M" arguments tell the Container to attempt to allocate 150 MiB of memory.Create the Pod:kubectl apply -f https://k8s.io/examples/pods/resource/pod-level-memory-request-limit.yaml --namespace=pod-resources-example Verify that the Pod is running:kubectl get pod memory-demo --namespace=pod-resources-example View detailed information about the Pod:kubectl get pod memory-demo --output=yaml --namespace=pod-resources-example The output shows that the Pod has a memory request of 100 MiB and a memory limit of 200 MiB.... spec: containers: ... resources: requests: memory: 100Mi limits: memory: 200Mi ... Run kubectl top to fetch the metrics for the pod:kubectl top pod memory-demo --namespace=pod-resources-example The output shows that the Pod is using about 162,900,000 bytes of memory, which is about 150 MiB. This is greater than the Pod's 100 MiB request, but within the Pod's 200 MiB limit.NAME CPU(cores) MEMORY(bytes) memory-demo <something> 162856960 Create a pod with CPU requests and limits at pod-levelTo specify a CPU request for a Pod, include the resources.requests.cpu field in the Pod spec manifest. To specify a CPU limit, include resources.limits.cpu.In this exercise, you create a Pod that has one container. The Pod has a request of 0.5 CPU and a limit of 1 CPU. Here is the configuration file for the Pod:pods/resource/pod-level-cpu-request-limit.yaml apiVersion: v1 kind: Pod metadata: name: cpu-demo namespace: pod-resources-example spec: resources: limits: cpu: "1" requests: cpu: "0.5" containers: - name: cpu-demo-ctr image: vish/stress args: - -cpus - "2" The args section of the configuration file provides arguments for the container when it starts. The -cpus "2" argument tells the Container to attempt to use 2 CPUs.Create the Pod:kubectl apply -f https://k8s.io/examples/pods/resource/pod-level-cpu-request-limit.yaml --namespace=pod-resources-example Verify that the Pod is running:kubectl get pod cpu-demo --namespace=pod-resources-example View detailed information about the Pod:kubectl get pod cpu-demo --output=yaml --namespace=pod-resources-example The output shows that the Pod has a CPU request of 500 milliCPU and a CPU limit of 1 CPU.spec: containers: ... resources: limits: cpu: "1" requests: cpu: 500m Use kubectl top to fetch the metrics for the Pod:kubectl top pod cpu-demo --namespace=pod-resources-example This example output shows that the Pod is using 974 milliCPU, which is slightly less than the limit of 1 CPU specified in the Pod configuration.NAME CPU(cores) MEMORY(bytes) cpu-demo 974m <something> Recall that by setting -cpu "2", you configured the Container to attempt to use 2 CPUs, but the Container is only being allowed to use about 1 CPU. The container's CPU use is being throttled, because the container is attempting to use more CPU resources than the Pod CPU limit.Create a pod with resource requests and limits at both pod-level and container-levelTo assign CPU and memory resources to a Pod, you can specify them at both the pod level and the container level. Include the resources field in the Pod spec to define resources for the entire Pod. Additionally, include the resources field within container's specification in the Pod's manifest to set container-specific resource requirements.In this exercise, you'll create a Pod with two containers to explore the interaction of pod-level and container-level resource specifications. The Pod itself will have defined CPU requests and limits, while only one of the containers will have its own explicit resource requests and limits. The other container will inherit the resource constraints from the pod-level settings. Here's the configuration file for the Pod:pods/resource/pod-level-resources.yaml apiVersion: v1 kind: Pod metadata: name: pod-resources-demo namespace: pod-resources-example spec: resources: limits: cpu: "1" memory: "200Mi" requests: cpu: "1" memory: "100Mi" containers: - name: pod-resources-demo-ctr-1 image: nginx resources: limits: cpu: "0.5" memory: "100Mi" requests: cpu: "0.5" memory: "50Mi" - name: pod-resources-demo-ctr-2 image: fedora command: - sleep - inf Create the Pod:kubectl apply -f https://k8s.io/examples/pods/resource/pod-level-resources.yaml --namespace=pod-resources-example Verify that the Pod Container is running:kubectl get pod-resources-demo --namespace=pod-resources-example View detailed information about the Pod:kubectl get pod memory-demo --output=yaml --namespace=pod-resources-example The output shows that one container in the Pod has a memory request of 50 MiB and a CPU request of 0.5 cores, with a memory limit of 100 MiB and a CPU limit of 0.5 cores. The Pod itself has a memory request of 100 MiB and a CPU request of 1 core, and a memory limit of 200 MiB and a CPU limit of 1 core.... containers: name: pod-resources-demo-ctr-1 resources: requests: cpu: 500m memory: 50Mi limits: cpu: 500m memory: 100Mi ... name: pod-resources-demo-ctr-2 resources: {} resources: limits: cpu: 1 memory: 200Mi requests: cpu: 1 memory: 100Mi ... Since pod-level requests and limits are specified, the request guarantees for both containers in the pod will be equal 1 core or CPU and 100Mi of memory. Additionally, both containers together won't be able to use more resources than specified in the pod-level limits, ensuring they cannot exceed a combined total of 200 MiB of memory and 1 core of CPU.Clean upDelete your namespace:kubectl delete namespace pod-resources-example What's nextFor application developersAssign Memory Resources to Containers and PodsAssign CPU Resources to Containers and PodsFor cluster administratorsConfigure Default Memory Requests and Limits for a NamespaceConfigure Default CPU Requests and Limits for a NamespaceConfigure Minimum and Maximum Memory Constraints for a NamespaceConfigure Minimum and Maximum CPU Constraints for a NamespaceConfigure Memory and CPU Quotas for a Namespace


Use an Image Volume With a PodFEATURE STATE: Kubernetes v1.33 [beta] (enabled by default: false)This page shows how to configure a pod using image volumes. This allows you to mount content from OCI registries inside containers.Before you beginYou need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. It is recommended to run this tutorial on a cluster with at least two nodes that are not acting as control plane hosts. If you do not already have a cluster, you can create one by using minikube or you can use one of these Kubernetes playgrounds:iximiuz LabsKillercodaKodeKloudPlay with KubernetesYour Kubernetes server must be at or later than version v1.31.To check the version, enter kubectl version.The container runtime needs to support the image volumes featureYou need to exec commands in the hostYou need to be able to exec into podsYou need to enable the ImageVolume feature gateRun a Pod that uses an image volumeAn image volume for a pod is enabled by setting the volumes.[*].image field of .spec to a valid reference and consuming it in the volumeMounts of the container. For example:pods/image-volumes.yaml apiVersion: v1 kind: Pod metadata: name: image-volume spec: containers: - name: shell command: ["sleep", "infinity"] image: debian volumeMounts: - name: volume mountPath: /volume volumes: - name: volume image: reference: quay.io/crio/artifact:v2 pullPolicy: IfNotPresent Create the pod on your cluster:kubectl apply -f https://k8s.io/examples/pods/image-volumes.yaml Attach to the container:kubectl attach -it image-volume bash Check the content of a file in the volume:cat /volume/dir/file The output is similar to:1 You can also check another file in a different path:cat /volume/file The output is similar to:2 Use subPath (or subPathExpr)It is possible to utilize subPath or subPathExpr from Kubernetes v1.33 when using the image volume feature.pods/image-volumes-subpath.yaml apiVersion: v1 kind: Pod metadata: name: image-volume spec: containers: - name: shell command: ["sleep", "infinity"] image: debian volumeMounts: - name: volume mountPath: /volume subPath: dir volumes: - name: volume image: reference: quay.io/crio/artifact:v2 pullPolicy: IfNotPresent Create the pod on your cluster:kubectl apply -f https://k8s.io/examples/pods/image-volumes-subpath.yaml Attach to the container:kubectl attach -it image-volume bash Check the content of the file from the dir sub path in the volume:cat /volume/file The output is similar to:1 Further readingimage volumes

Use an Image Volume With a Pod


PodTemplatePodTemplate describes a template for creating copies of a predefined pod.apiVersion: v1import "k8s.io/api/core/v1"PodTemplatePodTemplate describes a template for creating copies of a predefined pod.apiVersion: v1kind: PodTemplatemetadata (ObjectMeta)Standard object's metadata. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadatatemplate (PodTemplateSpec)Template defines the pods that will be created from this pod template. https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-statusPodTemplateSpecPodTemplateSpec describes the data a pod should have when created from a templatemetadata (ObjectMeta)Standard object's metadata. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadataspec (PodSpec)Specification of the desired behavior of the pod. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-statusPodTemplateListPodTemplateList is a list of PodTemplates.apiVersion: v1kind: PodTemplateListmetadata (ListMeta)Standard list metadata. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kindsitems ([]PodTemplate), requiredList of pod templatesOperationsget read the specified PodTemplateHTTP RequestGET /api/v1/namespaces/{namespace}/podtemplates/{name}Parametersname (in path): string, requiredname of the PodTemplatenamespace (in path): string, requirednamespacepretty (in query): stringprettyResponse200 (PodTemplate): OK401: Unauthorizedlist list or watch objects of kind PodTemplateHTTP RequestGET /api/v1/namespaces/{namespace}/podtemplatesParametersnamespace (in path): string, requirednamespaceallowWatchBookmarks (in query): booleanallowWatchBookmarkscontinue (in query): stringcontinuefieldSelector (in query): stringfieldSelectorlabelSelector (in query): stringlabelSelectorlimit (in query): integerlimitpretty (in query): stringprettyresourceVersion (in query): stringresourceVersionresourceVersionMatch (in query): stringresourceVersionMatchsendInitialEvents (in query): booleansendInitialEventstimeoutSeconds (in query): integertimeoutSecondswatch (in query): booleanwatchResponse200 (PodTemplateList): OK401: Unauthorizedlist list or watch objects of kind PodTemplateHTTP RequestGET /api/v1/podtemplatesParametersallowWatchBookmarks (in query): booleanallowWatchBookmarkscontinue (in query): stringcontinuefieldSelector (in query): stringfieldSelectorlabelSelector (in query): stringlabelSelectorlimit (in query): integerlimitpretty (in query): stringprettyresourceVersion (in query): stringresourceVersionresourceVersionMatch (in query): stringresourceVersionMatchsendInitialEvents (in query): booleansendInitialEventstimeoutSeconds (in query): integertimeoutSecondswatch (in query): booleanwatchResponse200 (PodTemplateList): OK401: Unauthorizedcreate create a PodTemplateHTTP RequestPOST /api/v1/namespaces/{namespace}/podtemplatesParametersnamespace (in path): string, requirednamespacebody: PodTemplate, requireddryRun (in query): stringdryRunfieldManager (in query): stringfieldManagerfieldValidation (in query): stringfieldValidationpretty (in query): stringprettyResponse200 (PodTemplate): OK201 (PodTemplate): Created202 (PodTemplate): Accepted401: Unauthorizedupdate replace the specified PodTemplateHTTP RequestPUT /api/v1/namespaces/{namespace}/podtemplates/{name}Parametersname (in path): string, requiredname of the PodTemplatenamespace (in path): string, requirednamespacebody: PodTemplate, requireddryRun (in query): stringdryRunfieldManager (in query): stringfieldManagerfieldValidation (in query): stringfieldValidationpretty (in query): stringprettyResponse200 (PodTemplate): OK201 (PodTemplate): Created401: Unauthorizedpatch partially update the specified PodTemplateHTTP RequestPATCH /api/v1/namespaces/{namespace}/podtemplates/{name}Parametersname (in path): string, requiredname of the PodTemplatenamespace (in path): string, requirednamespacebody: Patch, requireddryRun (in query): stringdryRunfieldManager (in query): stringfieldManagerfieldValidation (in query): stringfieldValidationforce (in query): booleanforcepretty (in query): stringprettyResponse200 (PodTemplate): OK201 (PodTemplate): Created401: Unauthorizeddelete delete a PodTemplateHTTP RequestDELETE /api/v1/namespaces/{namespace}/podtemplates/{name}Parametersname (in path): string, requiredname of the PodTemplatenamespace (in path): string, requirednamespacebody: DeleteOptionsdryRun (in query): stringdryRungracePeriodSeconds (in query): integergracePeriodSecondsignoreStoreReadErrorWithClusterBreakingPotential (in query): booleanignoreStoreReadErrorWithClusterBreakingPotentialpretty (in query): stringprettypropagationPolicy (in query): stringpropagationPolicyResponse200 (PodTemplate): OK202 (PodTemplate): Accepted401: Unauthorizeddeletecollection delete collection of PodTemplateHTTP RequestDELETE /api/v1/namespaces/{namespace}/podtemplatesParametersnamespace (in path): string, requirednamespacebody: DeleteOptionscontinue (in query): stringcontinuedryRun (in query): stringdryRunfieldSelector (in query): stringfieldSelectorgracePeriodSeconds (in query): integergracePeriodSecondsignoreStoreReadErrorWithClusterBreakingPotential (in query): booleanignoreStoreReadErrorWithClusterBreakingPotentiallabelSelector (in query): stringlabelSelectorlimit (in query): integerlimitpretty (in query): stringprettypropagationPolicy (in query): stringpropagationPolicyresourceVersion (in query): stringresourceVersionresourceVersionMatch (in query): stringresourceVersionMatchsendInitialEvents (in query): booleansendInitialEventstimeoutSeconds (in query): integertimeoutSecondsResponse200 (Status): OK401: UnauthorizedThis page is automatically generated.If you plan to report an issue with this page, mention that the page is auto-generated in your issue description. The fix may need to happen elsewhere in the Kubernetes project.

PodTemplate


Kubernetes Self-HealingKubernetes is designed with self-healing capabilities that help maintain the health and availability of workloads. It automatically replaces failed containers, reschedules workloads when nodes become unavailable, and ensures that the desired state of the system is maintained.Self-Healing capabilitiesContainer-level restarts: If a container inside a Pod fails, Kubernetes restarts it based on the restartPolicy.Replica replacement: If a Pod in a Deployment or StatefulSet fails, Kubernetes creates a replacement Pod to maintain the specified number of replicas. If a Pod fails that is part of a DaemonSet fails, the control plane creates a replacement Pod to run on the same node.Persistent storage recovery: If a node is running a Pod with a PersistentVolume (PV) attached, and the node fails, Kubernetes can reattach the volume to a new Pod on a different node.Load balancing for Services: If a Pod behind a Service fails, Kubernetes automatically removes it from the Service's endpoints to route traffic only to healthy Pods.Here are some of the key components that provide Kubernetes self-healing:kubelet: Ensures that containers are running, and restarts those that fail.ReplicaSet, StatefulSet and DaemonSet controller: Maintains the desired number of Pod replicas.PersistentVolume controller: Manages volume attachment and detachment for stateful workloads.ConsiderationsStorage Failures: If a persistent volume becomes unavailable, recovery steps may be required.Application Errors: Kubernetes can restart containers, but underlying application issues must be addressed separately.What's nextRead more about PodsLearn about Kubernetes ControllersExplore PersistentVolumesRead about node autoscaling. Node autoscaling also provides automatic healing if or when nodes fail in your cluster.

Kubernetes Self-Healing


Recommended LabelsYou can visualize and manage Kubernetes objects with more tools than kubectl and the dashboard. A common set of labels allows tools to work interoperably, describing objects in a common manner that all tools can understand.In addition to supporting tooling, the recommended labels describe applications in a way that can be queried.The metadata is organized around the concept of an application. Kubernetes is not a platform as a service (PaaS) and doesn't have or enforce a formal notion of an application. Instead, applications are informal and described with metadata. The definition of what an application contains is loose.Note:These are recommended labels. They make it easier to manage applications but aren't required for any core tooling.Shared labels and annotations share a common prefix: app.kubernetes.io. Labels without a prefix are private to users. The shared prefix ensures that shared labels do not interfere with custom user labels.LabelsIn order to take full advantage of using these labels, they should be applied on every resource object.KeyDescriptionExampleTypeapp.kubernetes.io/nameThe name of the applicationmysqlstringapp.kubernetes.io/instanceA unique name identifying the instance of an applicationmysql-abcxyzstringapp.kubernetes.io/versionThe current version of the application (e.g., a SemVer 1.0, revision hash, etc.)5.7.21stringapp.kubernetes.io/componentThe component within the architecturedatabasestringapp.kubernetes.io/part-ofThe name of a higher level application this one is part ofwordpressstringapp.kubernetes.io/managed-byThe tool being used to manage the operation of an applicationHelmstringTo illustrate these labels in action, consider the following StatefulSet object:# This is an excerpt apiVersion: apps/v1 kind: StatefulSet metadata: labels: app.kubernetes.io/name: mysql app.kubernetes.io/instance: mysql-abcxyz app.kubernetes.io/version: "5.7.21" app.kubernetes.io/component: database app.kubernetes.io/part-of: wordpress app.kubernetes.io/managed-by: Helm Applications And Instances Of ApplicationsAn application can be installed one or more times into a Kubernetes cluster and, in some cases, the same namespace. For example, WordPress can be installed more than once where different websites are different installations of WordPress.The name of an application and the instance name are recorded separately. For example, WordPress has a app.kubernetes.io/name of wordpress while it has an instance name, represented as app.kubernetes.io/instance with a value of wordpress-abcxyz. This enables the application and instance of the application to be identifiable. Every instance of an application must have a unique name.ExamplesTo illustrate different ways to use these labels the following examples have varying complexity.A Simple Stateless ServiceConsider the case for a simple stateless service deployed using Deployment and Service objects. The following two snippets represent how the labels could be used in their simplest form.The Deployment is used to oversee the pods running the application itself.apiVersion: apps/v1 kind: Deployment metadata: labels: app.kubernetes.io/name: myservice app.kubernetes.io/instance: myservice-abcxyz ... The Service is used to expose the application.apiVersion: v1 kind: Service metadata: labels: app.kubernetes.io/name: myservice app.kubernetes.io/instance: myservice-abcxyz ... Web Application With A DatabaseConsider a slightly more complicated application: a web application (WordPress) using a database (MySQL), installed using Helm. The following snippets illustrate the start of objects used to deploy this application.The start to the following Deployment is used for WordPress:apiVersion: apps/v1 kind: Deployment metadata: labels: app.kubernetes.io/name: wordpress app.kubernetes.io/instance: wordpress-abcxyz app.kubernetes.io/version: "4.9.4" app.kubernetes.io/managed-by: Helm app.kubernetes.io/component: server app.kubernetes.io/part-of: wordpress ... The Service is used to expose WordPress:apiVersion: v1 kind: Service metadata: labels: app.kubernetes.io/name: wordpress app.kubernetes.io/instance: wordpress-abcxyz app.kubernetes.io/version: "4.9.4" app.kubernetes.io/managed-by: Helm app.kubernetes.io/component: server app.kubernetes.io/part-of: wordpress ... MySQL is exposed as a StatefulSet with metadata for both it and the larger application it belongs to:apiVersion: apps/v1 kind: StatefulSet metadata: labels: app.kubernetes.io/name: mysql app.kubernetes.io/instance: mysql-abcxyz app.kubernetes.io/version: "5.7.21" app.kubernetes.io/managed-by: Helm app.kubernetes.io/component: database app.kubernetes.io/part-of: wordpress ... The Service is used to expose MySQL as part of WordPress:apiVersion: v1 kind: Service metadata: labels: app.kubernetes.io/name: mysql app.kubernetes.io/instance: mysql-abcxyz app.kubernetes.io/version: "5.7.21" app.kubernetes.io/managed-by: Helm app.kubernetes.io/component: database app.kubernetes.io/part-of: wordpress ... With the MySQL StatefulSet and Service you'll notice information about both MySQL and WordPress, the broader application, are included.

Recommended Labels


Administer a ClusterLearn common tasks for administering a cluster.Administration with kubeadmOverprovision Node Capacity For A ClusterMigrating from dockershimGenerate Certificates ManuallyManage Memory, CPU, and API ResourcesInstall a Network Policy ProviderAccess Clusters Using the Kubernetes APIAdvertise Extended Resources for a NodeAutoscale the DNS Service in a ClusterChange the Access Mode of a PersistentVolume to ReadWriteOncePodChange the default StorageClassSwitching from Polling to CRI Event-based Updates to Container StatusChange the Reclaim Policy of a PersistentVolumeCloud Controller Manager AdministrationConfigure a kubelet image credential providerConfigure Quotas for API ObjectsControl CPU Management Policies on the NodeControl Topology Management Policies on a nodeCustomizing DNS ServiceDebugging DNS ResolutionDeclare Network PolicyDeveloping Cloud Controller ManagerEnable Or Disable A Kubernetes APIEncrypting Confidential Data at RestDecrypt Confidential Data that is Already Encrypted at RestGuaranteed Scheduling For Critical Add-On PodsIP Masquerade Agent User GuideLimit Storage ConsumptionMigrate Replicated Control Plane To Use Cloud Controller ManagerOperating etcd clusters for KubernetesReserve Compute Resources for System DaemonsRunning Kubernetes Node Components as a Non-root UserSafely Drain a NodeSecuring a ClusterSet Kubelet Parameters Via A Configuration FileShare a Cluster with NamespacesUpgrade A ClusterUse Cascading Deletion in a ClusterUsing a KMS provider for data encryptionUsing CoreDNS for Service DiscoveryUsing NodeLocal DNSCache in Kubernetes ClustersUsing sysctls in a Kubernetes ClusterUtilizing the NUMA-aware Memory ManagerVerify Signed Kubernetes Artifacts

Administer a Cluster


Considerations for large clustersA cluster is a set of nodes (physical or virtual machines) running Kubernetes agents, managed by the control plane. Kubernetes v1.34 supports clusters with up to 5,000 nodes. More specifically, Kubernetes is designed to accommodate configurations that meet all of the following criteria:No more than 110 pods per nodeNo more than 5,000 nodesNo more than 150,000 total podsNo more than 300,000 total containersYou can scale your cluster by adding or removing nodes. The way you do this depends on how your cluster is deployed.Cloud provider resource quotasTo avoid running into cloud provider quota issues, when creating a cluster with many nodes, consider:Requesting a quota increase for cloud resources such as:Computer instancesCPUsStorage volumesIn-use IP addressesPacket filtering rule setsNumber of load balancersNetwork subnetsLog streamsGating the cluster scaling actions to bring up new nodes in batches, with a pause between batches, because some cloud providers rate limit the creation of new instances.Control plane componentsFor a large cluster, you need a control plane with sufficient compute and other resources.Typically you would run one or two control plane instances per failure zone, scaling those instances vertically first and then scaling horizontally after reaching the point of falling returns to (vertical) scale.You should run at least one instance per failure zone to provide fault-tolerance. Kubernetes nodes do not automatically steer traffic towards control-plane endpoints that are in the same failure zone; however, your cloud provider might have its own mechanisms to do this.For example, using a managed load balancer, you configure the load balancer to send traffic that originates from the kubelet and Pods in failure zone A, and direct that traffic only to the control plane hosts that are also in zone A. If a single control-plane host or endpoint failure zone A goes offline, that means that all the control-plane traffic for nodes in zone A is now being sent between zones. Running multiple control plane hosts in each zone makes that outcome less likely.etcd storageTo improve performance of large clusters, you can store Event objects in a separate dedicated etcd instance.When creating a cluster, you can (using custom tooling):start and configure additional etcd instanceconfigure the API server to use it for storing eventsSee Operating etcd clusters for Kubernetes and Set up a High Availability etcd cluster with kubeadm for details on configuring and managing etcd for a large cluster.Addon resourcesKubernetes resource limits help to minimize the impact of memory leaks and other ways that pods and containers can impact on other components. These resource limits apply to addon resources just as they apply to application workloads.For example, you can set CPU and memory limits for a logging component: ... containers: - name: fluentd-cloud-logging image: fluent/fluentd-kubernetes-daemonset:v1 resources: limits: cpu: 100m memory: 200Mi Addons' default limits are typically based on data collected from experience running each addon on small or medium Kubernetes clusters. When running on large clusters, addons often consume more of some resources than their default limits. If a large cluster is deployed without adjusting these values, the addon(s) may continuously get killed because they keep hitting the memory limit. Alternatively, the addon may run but with poor performance due to CPU time slice restrictions.To avoid running into cluster addon resource issues, when creating a cluster with many nodes, consider the following:Some addons scale vertically - there is one replica of the addon for the cluster or serving a whole failure zone. For these addons, increase requests and limits as you scale out your cluster.Many addons scale horizontally - you add capacity by running more pods - but with a very large cluster you may also need to raise CPU or memory limits slightly. The Vertical Pod Autoscaler can run in recommender mode to provide suggested figures for requests and limits.Some addons run as one copy per node, controlled by a DaemonSet: for example, a node-level log aggregator. Similar to the case with horizontally-scaled addons, you may also need to raise CPU or memory limits slightly.What's nextVerticalPodAutoscaler is a custom resource that you can deploy into your cluster to help you manage resource requests and limits for pods.Learn more about Vertical Pod Autoscaler and how you can use it to scale cluster components, including cluster-critical addons.Read about Node autoscalingThe addon resizer helps you in resizing the addons automatically as your cluster's scale changes.

Considerations for large clusters


User NamespacesFEATURE STATE: Kubernetes v1.30 [beta]This page explains how user namespaces are used in Kubernetes pods. A user namespace isolates the user running inside the container from the one in the host.A process running as root in a container can run as a different (non-root) user in the host; in other words, the process has full privileges for operations inside the user namespace, but is unprivileged for operations outside the namespace.You can use this feature to reduce the damage a compromised container can do to the host or other pods in the same node. There are several security vulnerabilities rated either HIGH or CRITICAL that were not exploitable when user namespaces is active. It is expected user namespace will mitigate some future vulnerabilities too.Before you beginNote: This section links to third party projects that provide functionality required by Kubernetes. The Kubernetes project authors aren't responsible for these projects, which are listed alphabetically. To add a project to this list, read the content guide before submitting a change. More information.This is a Linux-only feature and support is needed in Linux for idmap mounts on the filesystems used. This means:On the node, the filesystem you use for /var/lib/kubelet/pods/, or the custom directory you configure for this, needs idmap mount support.All the filesystems used in the pod's volumes must support idmap mounts.In practice this means you need at least Linux 6.3, as tmpfs started supporting idmap mounts in that version. This is usually needed as several Kubernetes features use tmpfs (the service account token that is mounted by default uses a tmpfs, Secrets use a tmpfs, etc.)Some popular filesystems that support idmap mounts in Linux 6.3 are: btrfs, ext4, xfs, fat, tmpfs, overlayfs.In addition, the container runtime and its underlying OCI runtime must support user namespaces. The following OCI runtimes offer support:crun version 1.9 or greater (it's recommend version 1.13+).runc version 1.2 or greaterNote:Some OCI runtimes do not include the support needed for using user namespaces in Linux pods. If you use a managed Kubernetes, or have downloaded it from packages and set it up, it's possible that nodes in your cluster use a runtime that doesn't include this support.To use user namespaces with Kubernetes, you also need to use a CRI container runtime to use this feature with Kubernetes pods:containerd: version 2.0 (and later) supports user namespaces for containers.CRI-O: version 1.25 (and later) supports user namespaces for containers.You can see the status of user namespaces support in cri-dockerd tracked in an issue on GitHub.IntroductionUser namespaces is a Linux feature that allows to map users in the container to different users in the host. Furthermore, the capabilities granted to a pod in a user namespace are valid only in the namespace and void outside of it.A pod can opt-in to use user namespaces by setting the pod.spec.hostUsers field to false.The kubelet will pick host UIDs/GIDs a pod is mapped to, and will do so in a way to guarantee that no two pods on the same node use the same mapping.The runAsUser, runAsGroup, fsGroup, etc. fields in the pod.spec always refer to the user inside the container. These users will be used for volume mounts (specified in pod.spec.volumes) and therefore the host UID/GID will not have any effect on writes/reads from volumes the pod can mount. In other words, the inodes created/read in volumes mounted by the pod will be the same as if the pod wasn't using user namespaces.This way, a pod can easily enable and disable user namespaces (without affecting its volume's file ownerships) and can also share volumes with pods without user namespaces by just setting the appropriate users inside the container (RunAsUser, RunAsGroup, fsGroup, etc.). This applies to any volume the pod can mount, including hostPath (if the pod is allowed to mount hostPath volumes).By default, the valid UIDs/GIDs when this feature is enabled is the range 0-65535. This applies to files and processes (runAsUser, runAsGroup, etc.).Files using a UID/GID outside this range will be seen as belonging to the overflow ID, usually 65534 (configured in /proc/sys/kernel/overflowuid and /proc/sys/kernel/overflowgid). However, it is not possible to modify those files, even by running as the 65534 user/group.If the range 0-65535 is extended with a configuration knob, the aforementioned restrictions apply to the extended range.Most applications that need to run as root but don't access other host namespaces or resources, should continue to run fine without any changes needed if user namespaces is activated.Understanding user namespaces for podsSeveral container runtimes with their default configuration (like Docker Engine, containerd, CRI-O) use Linux namespaces for isolation. Other technologies exist and can be used with those runtimes too (e.g. Kata Containers uses VMs instead of Linux namespaces). This page is applicable for container runtimes using Linux namespaces for isolation.When creating a pod, by default, several new namespaces are used for isolation: a network namespace to isolate the network of the container, a PID namespace to isolate the view of processes, etc. If a user namespace is used, this will isolate the users in the container from the users in the node.This means containers can run as root and be mapped to a non-root user on the host. Inside the container the process will think it is running as root (and therefore tools like apt, yum, etc. work fine), while in reality the process doesn't have privileges on the host. You can verify this, for example, if you check which user the container process is running by executing ps aux from the host. The user ps shows is not the same as the user you see if you execute inside the container the command id.This abstraction limits what can happen, for example, if the container manages to escape to the host. Given that the container is running as a non-privileged user on the host, it is limited what it can do to the host.Furthermore, as users on each pod will be mapped to different non-overlapping users in the host, it is limited what they can do to other pods too.Capabilities granted to a pod are also limited to the pod user namespace and mostly invalid out of it, some are even completely void. Here are two examples:CAP_SYS_MODULE does not have any effect if granted to a pod using user namespaces, the pod isn't able to load kernel modules.CAP_SYS_ADMIN is limited to the pod's user namespace and invalid outside of it.Without using a user namespace a container running as root, in the case of a container breakout, has root privileges on the node. And if some capability were granted to the container, the capabilities are valid on the host too. None of this is true when we use user namespaces.If you want to know more details about what changes when user namespaces are in use, see man 7 user_namespaces.Set up a node to support user namespacesBy default, the kubelet assigns pods UIDs/GIDs above the range 0-65535, based on the assumption that the host's files and processes use UIDs/GIDs within this range, which is standard for most Linux distributions. This approach prevents any overlap between the UIDs/GIDs of the host and those of the pods.Avoiding the overlap is important to mitigate the impact of vulnerabilities such as CVE-2021-25741, where a pod can potentially read arbitrary files in the host. If the UIDs/GIDs of the pod and the host don't overlap, it is limited what a pod would be able to do: the pod UID/GID won't match the host's file owner/group.The kubelet can use a custom range for user IDs and group IDs for pods. To configure a custom range, the node needs to have:A user kubelet in the system (you cannot use any other username here)The binary getsubids installed (part of shadow-utils) and in the PATH for the kubelet binary.A configuration of subordinate UIDs/GIDs for the kubelet user (see man 5 subuid and man 5 subgid).This setting only gathers the UID/GID range configuration and does not change the user executing the kubelet.You must follow some constraints for the subordinate ID range that you assign to the kubelet user:The subordinate user ID, that starts the UID range for Pods, must be a multiple of 65536 and must also be greater than or equal to 65536. In other words, you cannot use any ID from the range 0-65535 for Pods; the kubelet imposes this restriction to make it difficult to create an accidentally insecure configuration.The subordinate ID count must be a multiple of 65536The subordinate ID count must be at least 65536 x <maxPods> where <maxPods> is the maximum number of pods that can run on the node.You must assign the same range for both user IDs and for group IDs, It doesn't matter if other users have user ID ranges that don't align with the group ID ranges.None of the assigned ranges should overlap with any other assignment.The subordinate configuration must be only one line. In other words, you can't have multiple ranges.For example, you could define /etc/subuid and /etc/subgid to both have these entries for the kubelet user:# The format is # name:firstID:count of IDs # where # - firstID is 65536 (the minimum value possible) # - count of IDs is 110 * 65536 # (110 is the default limit for number of pods on the node) kubelet:65536:7208960 ID count for each of PodsStarting with Kubernetes v1.33, the ID count for each of Pods can be set in KubeletConfiguration.apiVersion: kubelet.config.k8s.io/v1beta1 kind: KubeletConfiguration userNamespaces: idsPerPod: 1048576 The value of idsPerPod (uint32) must be a multiple of 65536. The default value is 65536. This value only applies to containers created after the kubelet was started with this KubeletConfiguration. Running containers are not affected by this config.In Kubernetes prior to v1.33, the ID count for each of Pods was hard-coded to 65536.Integration with Pod security admission checksFEATURE STATE: Kubernetes v1.29 [alpha]For Linux Pods that enable user namespaces, Kubernetes relaxes the application of Pod Security Standards in a controlled way. This behavior can be controlled by the feature gate UserNamespacesPodSecurityStandards, which allows an early opt-in for end users. Admins have to ensure that user namespaces are enabled by all nodes within the cluster if using the feature gate.If you enable the associated feature gate and create a Pod that uses user namespaces, the following fields won't be constrained even in contexts that enforce the Baseline or Restricted pod security standard. This behavior does not present a security concern because root inside a Pod with user namespaces actually refers to the user inside the container, that is never mapped to a privileged user on the host. Here's the list of fields that are not checks for Pods in those circumstances:spec.securityContext.runAsNonRootspec.containers[*].securityContext.runAsNonRootspec.initContainers[*].securityContext.runAsNonRootspec.ephemeralContainers[*].securityContext.runAsNonRootspec.securityContext.runAsUserspec.containers[*].securityContext.runAsUserspec.initContainers[*].securityContext.runAsUserspec.ephemeralContainers[*].securityContext.runAsUserLimitationsWhen using a user namespace for the pod, it is disallowed to use other host namespaces. In particular, if you set hostUsers: false then you are not allowed to set any of:hostNetwork: truehostIPC: truehostPID: trueNo container can use volumeDevices (raw block volumes, like /dev/sda) either. This includes all the container arrays in the pod spec:containersinitContainersephemeralContainersMetrics and observabilityThe kubelet exports two prometheus metrics specific to user-namespaces:started_user_namespaced_pods_total: a counter that tracks the number of user namespaced pods that are attempted to be created.started_user_namespaced_pods_errors_total: a counter that tracks the number of errors creating user namespaced pods.What's nextTake a look at Use a User Namespace With a PodItems on this page refer to third party products or projects that provide functionality required by Kubernetes. The Kubernetes project authors aren't responsible for those third-party products or projects. See the CNCF website guidelines for more details.You should read the content guide before proposing a change that adds an extra third-party link.

User Namespaces


Security For Windows NodesThis page describes security considerations and best practices specific to the Windows operating system.Protection for Secret data on nodesOn Windows, data from Secrets are written out in clear text onto the node's local storage (as compared to using tmpfs / in-memory filesystems on Linux). As a cluster operator, you should take both of the following additional measures:Use file ACLs to secure the Secrets' file location.Apply volume-level encryption using BitLocker.Container usersRunAsUsername can be specified for Windows Pods or containers to execute the container processes as specific user. This is roughly equivalent to RunAsUser.Windows containers offer two default user accounts, ContainerUser and ContainerAdministrator. The differences between these two user accounts are covered in When to use ContainerAdmin and ContainerUser user accounts within Microsoft's Secure Windows containers documentation.Local users can be added to container images during the container build process.Note:Nano Server based images run as ContainerUser by defaultServer Core based images run as ContainerAdministrator by defaultWindows containers can also run as Active Directory identities by utilizing Group Managed Service AccountsPod-level security isolationLinux-specific pod security context mechanisms (such as SELinux, AppArmor, Seccomp, or custom POSIX capabilities) are not supported on Windows nodes.Privileged containers are not supported on Windows. Instead HostProcess containers can be used on Windows to perform many of the tasks performed by privileged containers on Linux.

Security For Windows Nodes


Troubleshooting CNI plugin-related errorsTo avoid CNI plugin-related errors, verify that you are using or upgrading to a container runtime that has been tested to work correctly with your version of Kubernetes.About the "Incompatible CNI versions" and "Failed to destroy network for sandbox" errorsService issues exist for pod CNI network setup and tear down in containerd v1.6.0-v1.6.3 when the CNI plugins have not been upgraded and/or the CNI config version is not declared in the CNI config files. The containerd team reports, "these issues are resolved in containerd v1.6.4."With containerd v1.6.0-v1.6.3, if you do not upgrade the CNI plugins and/or declare the CNI config version, you might encounter the following "Incompatible CNI versions" or "Failed to destroy network for sandbox" error conditions.Incompatible CNI versions errorIf the version of your CNI plugin does not correctly match the plugin version in the config because the config version is later than the plugin version, the containerd log will likely show an error message on startup of a pod similar to:incompatible CNI versions; config is \"1.0.0\", plugin supports [\"0.1.0\" \"0.2.0\" \"0.3.0\" \"0.3.1\" \"0.4.0\"]" To fix this issue, update your CNI plugins and CNI config files.Failed to destroy network for sandbox errorIf the version of the plugin is missing in the CNI plugin config, the pod may run. However, stopping the pod generates an error similar to:ERROR[2022-04-26T00:43:24.518165483Z] StopPodSandbox for "b" failed error="failed to destroy network for sandbox \"bbc85f891eaf060c5a879e27bba9b6b06450210161dfdecfbb2732959fb6500a\": invalid version \"\": the version is empty" This error leaves the pod in the not-ready state with a network namespace still attached. To recover from this problem, edit the CNI config file to add the missing version information. The next attempt to stop the pod should be successful.Updating your CNI plugins and CNI config filesIf you're using containerd v1.6.0-v1.6.3 and encountered "Incompatible CNI versions" or "Failed to destroy network for sandbox" errors, consider updating your CNI plugins and editing the CNI config files.Here's an overview of the typical steps for each node:Safely drain and cordon the node.After stopping your container runtime and kubelet services, perform the following upgrade operations:If you're running CNI plugins, upgrade them to the latest version.If you're using non-CNI plugins, replace them with CNI plugins. Use the latest version of the plugins.Update the plugin configuration file to specify or match a version of the CNI specification that the plugin supports, as shown in the following "An example containerd configuration file" section.For containerd, ensure that you have installed the latest version (v1.0.0 or later) of the CNI loopback plugin.Upgrade node components (for example, the kubelet) to Kubernetes v1.24Upgrade to or install the most current version of the container runtime.Bring the node back into your cluster by restarting your container runtime and kubelet. Uncordon the node (kubectl uncordon <nodename>).An example containerd configuration fileThe following example shows a configuration for containerd runtime v1.6.x, which supports a recent version of the CNI specification (v1.0.0).Please see the documentation from your plugin and networking provider for further instructions on configuring your system.On Kubernetes, containerd runtime adds a loopback interface, lo, to pods as a default behavior. The containerd runtime configures the loopback interface via a CNI plugin, loopback. The loopback plugin is distributed as part of the containerd release packages that have the cni designation. containerd v1.6.0 and later includes a CNI v1.0.0-compatible loopback plugin as well as other default CNI plugins. The configuration for the loopback plugin is done internally by containerd, and is set to use CNI v1.0.0. This also means that the version of the loopback plugin must be v1.0.0 or later when this newer version containerd is started.The following bash command generates an example CNI config. Here, the 1.0.0 value for the config version is assigned to the cniVersion field for use when containerd invokes the CNI bridge plugin.cat << EOF | tee /etc/cni/net.d/10-containerd-net.conflist { "cniVersion": "1.0.0", "name": "containerd-net", "plugins": [ { "type": "bridge", "bridge": "cni0", "isGateway": true, "ipMasq": true, "promiscMode": true, "ipam": { "type": "host-local", "ranges": [ [{ "subnet": "10.88.0.0/16" }], [{ "subnet": "2001:db8:4860::/64" }] ], "routes": [ { "dst": "0.0.0.0/0" }, { "dst": "::/0" } ] } }, { "type": "portmap", "capabilities": {"portMappings": true}, "externalSetMarkChain": "KUBE-MARK-MASQ" } ] } EOF Update the IP address ranges in the preceding example with ones that are based on your use case and network addressing plan.


Indexed Job for Parallel Processing with Static Work AssignmentFEATURE STATE: Kubernetes v1.24 [stable]In this example, you will run a Kubernetes Job that uses multiple parallel worker processes. Each worker is a different container running in its own Pod. The Pods have an index number that the control plane sets automatically, which allows each Pod to identify which part of the overall task to work on.The pod index is available in the annotation batch.kubernetes.io/job-completion-index as a string representing its decimal value. In order for the containerized task process to obtain this index, you can publish the value of the annotation using the downward API mechanism. For convenience, the control plane automatically sets the downward API to expose the index in the JOB_COMPLETION_INDEX environment variable.Here is an overview of the steps in this example:Define a Job manifest using indexed completion. The downward API allows you to pass the pod index annotation as an environment variable or file to the container.Start an Indexed Job based on that manifest.Before you beginYou should already be familiar with the basic, non-parallel, use of Job.You need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. It is recommended to run this tutorial on a cluster with at least two nodes that are not acting as control plane hosts. If you do not already have a cluster, you can create one by using minikube or you can use one of these Kubernetes playgrounds:iximiuz LabsKillercodaKodeKloudPlay with KubernetesYour Kubernetes server must be at or later than version v1.21.To check the version, enter kubectl version.Choose an approachTo access the work item from the worker program, you have a few options:Read the JOB_COMPLETION_INDEX environment variable. The Job controller automatically links this variable to the annotation containing the completion index.Read a file that contains the completion index.Assuming that you can't modify the program, you can wrap it with a script that reads the index using any of the methods above and converts it into something that the program can use as input.For this example, imagine that you chose option 3 and you want to run the rev utility. This program accepts a file as an argument and prints its content reversed.rev data.txt You'll use the rev tool from the busybox container image.As this is only an example, each Pod only does a tiny piece of work (reversing a short string). In a real workload you might, for example, create a Job that represents the task of producing 60 seconds of video based on scene data. Each work item in the video rendering Job would be to render a particular frame of that video clip. Indexed completion would mean that each Pod in the Job knows which frame to render and publish, by counting frames from the start of the clip.Define an Indexed JobHere is a sample Job manifest that uses Indexed completion mode:application/job/indexed-job.yaml apiVersion: batch/v1 kind: Job metadata: name: 'indexed-job' spec: completions: 5 parallelism: 3 completionMode: Indexed template: spec: restartPolicy: Never initContainers: - name: 'input' image: 'docker.io/library/bash' command: - "bash" - "-c" - | items=(foo bar baz qux xyz) echo ${items[$JOB_COMPLETION_INDEX]} > /input/data.txt volumeMounts: - mountPath: /input name: input containers: - name: 'worker' image: 'docker.io/library/busybox' command: - "rev" - "/input/data.txt" volumeMounts: - mountPath: /input name: input volumes: - name: input emptyDir: {} In the example above, you use the builtin JOB_COMPLETION_INDEX environment variable set by the Job controller for all containers. An init container maps the index to a static value and writes it to a file that is shared with the container running the worker through an emptyDir volume. Optionally, you can define your own environment variable through the downward API to publish the index to containers. You can also choose to load a list of values from a ConfigMap as an environment variable or file.Alternatively, you can directly use the downward API to pass the annotation value as a volume file, like shown in the following example:application/job/indexed-job-vol.yaml apiVersion: batch/v1 kind: Job metadata: name: 'indexed-job' spec: completions: 5 parallelism: 3 completionMode: Indexed template: spec: restartPolicy: Never containers: - name: 'worker' image: 'docker.io/library/busybox' command: - "rev" - "/input/data.txt" volumeMounts: - mountPath: /input name: input volumes: - name: input downwardAPI: items: - path: "data.txt" fieldRef: fieldPath: metadata.annotations['batch.kubernetes.io/job-completion-index']Running the JobNow run the Job:# This uses the first approach (relying on $JOB_COMPLETION_INDEX) kubectl apply -f https://kubernetes.io/examples/application/job/indexed-job.yaml When you create this Job, the control plane creates a series of Pods, one for each index you specified. The value of .spec.parallelism determines how many can run at once whereas .spec.completions determines how many Pods the Job creates in total.Because .spec.parallelism is less than .spec.completions, the control plane waits for some of the first Pods to complete before starting more of them.You can wait for the Job to succeed, with a timeout:# The check for condition name is case insensitive kubectl wait --for=condition=complete --timeout=300s job/indexed-job Now, describe the Job and check that it was successful.kubectl describe jobs/indexed-job The output is similar to:Name: indexed-job Namespace: default Selector: controller-uid=bf865e04-0b67-483b-9a90-74cfc4c3e756 Labels: controller-uid=bf865e04-0b67-483b-9a90-74cfc4c3e756 job-name=indexed-job Annotations: <none> Parallelism: 3 Completions: 5 Start Time: Thu, 11 Mar 2021 15:47:34 +0000 Pods Statuses: 2 Running / 3 Succeeded / 0 Failed Completed Indexes: 0-2 Pod Template: Labels: controller-uid=bf865e04-0b67-483b-9a90-74cfc4c3e756 job-name=indexed-job Init Containers: input: Image: docker.io/library/bash Port: <none> Host Port: <none> Command: bash -c items=(foo bar baz qux xyz) echo ${items[$JOB_COMPLETION_INDEX]} > /input/data.txt Environment: <none> Mounts: /input from input (rw) Containers: worker: Image: docker.io/library/busybox Port: <none> Host Port: <none> Command: rev /input/data.txt Environment: <none> Mounts: /input from input (rw) Volumes: input: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: SizeLimit: <unset> Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal SuccessfulCreate 4s job-controller Created pod: indexed-job-njkjj Normal SuccessfulCreate 4s job-controller Created pod: indexed-job-9kd4h Normal SuccessfulCreate 4s job-controller Created pod: indexed-job-qjwsz Normal SuccessfulCreate 1s job-controller Created pod: indexed-job-fdhq5 Normal SuccessfulCreate 1s job-controller Created pod: indexed-job-ncslj In this example, you run the Job with custom values for each index. You can inspect the output of one of the pods:kubectl logs indexed-job-fdhq5 # Change this to match the name of a Pod from that Job The output is similar to:xuq