# Network Policies

Network PoliciesIf you want to control traffic flow at the IP address or port level (OSI layer 3 or 4), NetworkPolicies allow you to specify rules for traffic flow within your cluster, and also between Pods and the outside world. Your cluster must use a network plugin that supports NetworkPolicy enforcement.If you want to control traffic flow at the IP address or port level for TCP, UDP, and SCTP protocols, then you might consider using Kubernetes NetworkPolicies for particular applications in your cluster. NetworkPolicies are an application-centric construct which allow you to specify how a pod is allowed to communicate with various network "entities" (we use the word "entity" here to avoid overloading the more common terms such as "endpoints" and "services", which have specific Kubernetes connotations) over the network. NetworkPolicies apply to a connection with a pod on one or both ends, and are not relevant to other connections.The entities that a Pod can communicate with are identified through a combination of the following three identifiers:Other pods that are allowed (exception: a pod cannot block access to itself)Namespaces that are allowedIP blocks (exception: traffic to and from the node where a Pod is running is always allowed, regardless of the IP address of the Pod or the node)When defining a pod- or namespace-based NetworkPolicy, you use a selector to specify what traffic is allowed to and from the Pod(s) that match the selector.Meanwhile, when IP-based NetworkPolicies are created, we define policies based on IP blocks (CIDR ranges).PrerequisitesNetwork policies are implemented by the network plugin. To use network policies, you must be using a networking solution which supports NetworkPolicy. Creating a NetworkPolicy resource without a controller that implements it will have no effect.The two sorts of pod isolationThere are two sorts of isolation for a pod: isolation for egress, and isolation for ingress. They concern what connections may be established. "Isolation" here is not absolute, rather it means "some restrictions apply". The alternative, "non-isolated for $direction", means that no restrictions apply in the stated direction. The two sorts of isolation (or not) are declared independently, and are both relevant for a connection from one pod to another.By default, a pod is non-isolated for egress; all outbound connections are allowed. A pod is isolated for egress if there is any NetworkPolicy that both selects the pod and has "Egress" in its policyTypes; we say that such a policy applies to the pod for egress. When a pod is isolated for egress, the only allowed connections from the pod are those allowed by the egress list of some NetworkPolicy that applies to the pod for egress. Reply traffic for those allowed connections will also be implicitly allowed. The effects of those egress lists combine additively.By default, a pod is non-isolated for ingress; all inbound connections are allowed. A pod is isolated for ingress if there is any NetworkPolicy that both selects the pod and has "Ingress" in its policyTypes; we say that such a policy applies to the pod for ingress. When a pod is isolated for ingress, the only allowed connections into the pod are those from the pod's node and those allowed by the ingress list of some NetworkPolicy that applies to the pod for ingress. Reply traffic for those allowed connections will also be implicitly allowed. The effects of those ingress lists combine additively.Network policies do not conflict; they are additive. If any policy or policies apply to a given pod for a given direction, the connections allowed in that direction from that pod is the union of what the applicable policies allow. Thus, order of evaluation does not affect the policy result.For a connection from a source pod to a destination pod to be allowed, both the egress policy on the source pod and the ingress policy on the destination pod need to allow the connection. If either side does not allow the connection, it will not happen.The NetworkPolicy resourceSee the NetworkPolicy reference for a full definition of the resource.An example NetworkPolicy might look like this:service/networking/networkpolicy.yaml apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: test-network-policy namespace: default spec: podSelector: matchLabels: role: db policyTypes: - Ingress - Egress ingress: - from: - ipBlock: cidr: 172.17.0.0/16 except: - 172.17.1.0/24 - namespaceSelector: matchLabels: project: myproject - podSelector: matchLabels: role: frontend ports: - protocol: TCP port: 6379 egress: - to: - ipBlock: cidr: 10.0.0.0/24 ports: - protocol: TCP port: 5978 Note:POSTing this to the API server for your cluster will have no effect unless your chosen networking solution supports network policy.Mandatory Fields: As with all other Kubernetes config, a NetworkPolicy needs apiVersion, kind, and metadata fields. For general information about working with config files, see Configure a Pod to Use a ConfigMap, and Object Management.spec: NetworkPolicy spec has all the information needed to define a particular network policy in the given namespace.podSelector: Each NetworkPolicy includes a podSelector which selects the grouping of pods to which the policy applies. The example policy selects pods with the label "role=db". An empty podSelector selects all pods in the namespace.policyTypes: Each NetworkPolicy includes a policyTypes list which may include either Ingress, Egress, or both. The policyTypes field indicates whether or not the given policy applies to ingress traffic to selected pod, egress traffic from selected pods, or both. If no policyTypes are specified on a NetworkPolicy then by default Ingress will always be set and Egress will be set if the NetworkPolicy has any egress rules.ingress: Each NetworkPolicy may include a list of allowed ingress rules. Each rule allows traffic which matches both the from and ports sections. The example policy contains a single rule, which matches traffic on a single port, from one of three sources, the first specified via an ipBlock, the second via a namespaceSelector and the third via a podSelector.egress: Each NetworkPolicy may include a list of allowed egress rules. Each rule allows traffic which matches both the to and ports sections. The example policy contains a single rule, which matches traffic on a single port to any destination in 10.0.0.0/24.So, the example NetworkPolicy:isolates role=db pods in the default namespace for both ingress and egress traffic (if they weren't already isolated)(Ingress rules) allows connections to all pods in the default namespace with the label role=db on TCP port 6379 from:any pod in the default namespace with the label role=frontendany pod in a namespace with the label project=myprojectIP addresses in the ranges 172.17.0.0–172.17.0.255 and 172.17.2.0–172.17.255.255 (ie, all of 172.17.0.0/16 except 172.17.1.0/24)(Egress rules) allows connections from any pod in the default namespace with the label role=db to CIDR 10.0.0.0/24 on TCP port 5978See the Declare Network Policy walkthrough for further examples.Behavior of to and from selectorsThere are four kinds of selectors that can be specified in an ingress from section or egress to section:podSelector: This selects particular Pods in the same namespace as the NetworkPolicy which should be allowed as ingress sources or egress destinations.namespaceSelector: This selects particular namespaces for which all Pods should be allowed as ingress sources or egress destinations.namespaceSelector and podSelector: A single to/from entry that specifies both namespaceSelector and podSelector selects particular Pods within particular namespaces. Be careful to use correct YAML syntax. For example: ... ingress: - from: - namespaceSelector: matchLabels: user: alice podSelector: matchLabels: role: client ... This policy contains a single from element allowing connections from Pods with the label role=client in namespaces with the label user=alice. But the following policy is different: ... ingress: - from: - namespaceSelector: matchLabels: user: alice - podSelector: matchLabels: role: client ... It contains two elements in the from array, and allows connections from Pods in the local Namespace with the label role=client, or from any Pod in any namespace with the label user=alice.When in doubt, use kubectl describe to see how Kubernetes has interpreted the policy.ipBlock: This selects particular IP CIDR ranges to allow as ingress sources or egress destinations. These should be cluster-external IPs, since Pod IPs are ephemeral and unpredictable.Cluster ingress and egress mechanisms often require rewriting the source or destination IP of packets. In cases where this happens, it is not defined whether this happens before or after NetworkPolicy processing, and the behavior may be different for different combinations of network plugin, cloud provider, Service implementation, etc.In the case of ingress, this means that in some cases you may be able to filter incoming packets based on the actual original source IP, while in other cases, the "source IP" that the NetworkPolicy acts on may be the IP of a LoadBalancer or of the Pod's node, etc.For egress, this means that connections from pods to Service IPs that get rewritten to cluster-external IPs may or may not be subject to ipBlock-based policies.Default policiesBy default, if no policies exist in a namespace, then all ingress and egress traffic is allowed to and from pods in that namespace. The following examples let you change the default behavior in that namespace.Default deny all ingress trafficYou can create a "default" ingress isolation policy for a namespace by creating a NetworkPolicy that selects all pods but does not allow any ingress traffic to those pods.service/networking/network-policy-default-deny-ingress.yaml --- apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: default-deny-ingress spec: podSelector: {} policyTypes: - Ingress This ensures that even pods that aren't selected by any other NetworkPolicy will still be isolated for ingress. This policy does not affect isolation for egress from any pod.Allow all ingress trafficIf you want to allow all incoming connections to all pods in a namespace, you can create a policy that explicitly allows that.service/networking/network-policy-allow-all-ingress.yaml --- apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: allow-all-ingress spec: podSelector: {} ingress: - {} policyTypes: - Ingress With this policy in place, no additional policy or policies can cause any incoming connection to those pods to be denied. This policy has no effect on isolation for egress from any pod.Default deny all egress trafficYou can create a "default" egress isolation policy for a namespace by creating a NetworkPolicy that selects all pods but does not allow any egress traffic from those pods.service/networking/network-policy-default-deny-egress.yaml --- apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: default-deny-egress spec: podSelector: {} policyTypes: - Egress This ensures that even pods that aren't selected by any other NetworkPolicy will not be allowed egress traffic. This policy does not change the ingress isolation behavior of any pod.Allow all egress trafficIf you want to allow all connections from all pods in a namespace, you can create a policy that explicitly allows all outgoing connections from pods in that namespace.service/networking/network-policy-allow-all-egress.yaml --- apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: allow-all-egress spec: podSelector: {} egress: - {} policyTypes: - Egress With this policy in place, no additional policy or policies can cause any outgoing connection from those pods to be denied. This policy has no effect on isolation for ingress to any pod.Default deny all ingress and all egress trafficYou can create a "default" policy for a namespace which prevents all ingress AND egress traffic by creating the following NetworkPolicy in that namespace.service/networking/network-policy-default-deny-all.yaml --- apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: default-deny-all spec: podSelector: {} policyTypes: - Ingress - Egress This ensures that even pods that aren't selected by any other NetworkPolicy will not be allowed ingress or egress traffic.Network traffic filteringNetworkPolicy is defined for layer 4 connections (TCP, UDP, and optionally SCTP). For all the other protocols, the behaviour may vary across network plugins.Note:You must be using a CNI plugin that supports SCTP protocol NetworkPolicies.When a deny all network policy is defined, it is only guaranteed to deny TCP, UDP and SCTP connections. For other protocols, such as ARP or ICMP, the behaviour is undefined. The same applies to allow rules: when a specific pod is allowed as ingress source or egress destination, it is undefined what happens with (for example) ICMP packets. Protocols such as ICMP may be allowed by some network plugins and denied by others.Targeting a range of portsFEATURE STATE: Kubernetes v1.25 [stable]When writing a NetworkPolicy, you can target a range of ports instead of a single port.This is achievable with the usage of the endPort field, as the following example:service/networking/networkpolicy-multiport-egress.yaml apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: multi-port-egress namespace: default spec: podSelector: matchLabels: role: db policyTypes: - Egress egress: - to: - ipBlock: cidr: 10.0.0.0/24 ports: - protocol: TCP port: 32000 endPort: 32768 The above rule allows any Pod with label role=db on the namespace default to communicate with any IP within the range 10.0.0.0/24 over TCP, provided that the target port is between the range 32000 and 32768.The following restrictions apply when using this field:The endPort field must be equal to or greater than the port field.endPort can only be defined if port is also defined.Both ports must be numeric.Note:Your cluster must be using a CNI plugin that supports the endPort field in NetworkPolicy specifications. If your network plugin does not support the endPort field and you specify a NetworkPolicy with that, the policy will be applied only for the single port field.Targeting multiple namespaces by labelIn this scenario, your Egress NetworkPolicy targets more than one namespace using their label names. For this to work, you need to label the target namespaces. For example:kubectl label namespace frontend namespace=frontend kubectl label namespace backend namespace=backend Add the labels under namespaceSelector in your NetworkPolicy document. For example:apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: egress-namespaces spec: podSelector: matchLabels: app: myapp policyTypes: - Egress egress: - to: - namespaceSelector: matchExpressions: - key: namespace operator: In values: ["frontend", "backend"] Note:It is not possible to directly specify the name of the namespaces in a NetworkPolicy. You must use a namespaceSelector with matchLabels or matchExpressions to select the namespaces based on their labels.Targeting a Namespace by its nameThe Kubernetes control plane sets an immutable label kubernetes.io/metadata.name on all namespaces, the value of the label is the namespace name.While NetworkPolicy cannot target a namespace by its name with some object field, you can use the standardized label to target a specific namespace.Pod lifecycleNote:The following applies to clusters with a conformant networking plugin and a conformant implementation of NetworkPolicy.When a new NetworkPolicy object is created, it may take some time for a network plugin to handle the new object. If a pod that is affected by a NetworkPolicy is created before the network plugin has completed NetworkPolicy handling, that pod may be started unprotected, and isolation rules will be applied when the NetworkPolicy handling is completed.Once the NetworkPolicy is handled by a network plugin,All newly created pods affected by a given NetworkPolicy will be isolated before they are started. Implementations of NetworkPolicy must ensure that filtering is effective throughout the Pod lifecycle, even from the very first instant that any container in that Pod is started. Because they are applied at Pod level, NetworkPolicies apply equally to init containers, sidecar containers, and regular containers.Allow rules will be applied eventually after the isolation rules (or may be applied at the same time). In the worst case, a newly created pod may have no network connectivity at all when it is first started, if isolation rules were already applied, but no allow rules were applied yet.Every created NetworkPolicy will be handled by a network plugin eventually, but there is no way to tell from the Kubernetes API when exactly that happens.Therefore, pods must be resilient against being started up with different network connectivity than expected. If you need to make sure the pod can reach certain destinations before being started, you can use an init container to wait for those destinations to be reachable before kubelet starts the app containers.Every NetworkPolicy will be applied to all selected pods eventually. Because the network plugin may implement NetworkPolicy in a distributed manner, it is possible that pods may see a slightly inconsistent view of network policies when the pod is first created, or when pods or policies change. For example, a newly-created pod that is supposed to be able to reach both Pod A on Node 1 and Pod B on Node 2 may find that it can reach Pod A immediately, but cannot reach Pod B until a few seconds later.NetworkPolicy and hostNetwork podsNetworkPolicy behaviour for hostNetwork pods is undefined, but it should be limited to 2 possibilities:The network plugin can distinguish hostNetwork pod traffic from all other traffic (including being able to distinguish traffic from different hostNetwork pods on the same node), and will apply NetworkPolicy to hostNetwork pods just like it does to pod-network pods.The network plugin cannot properly distinguish hostNetwork pod traffic, and so it ignores hostNetwork pods when matching podSelector and namespaceSelector. Traffic to/from hostNetwork pods is treated the same as all other traffic to/from the node IP. (This is the most common implementation.)This applies whena hostNetwork pod is selected by spec.podSelector. ... spec: podSelector: matchLabels: role: client ... a hostNetwork pod is selected by a podSelector or namespaceSelector in an ingress or egress rule. ... ingress: - from: - podSelector: matchLabels: role: client ... At the same time, since hostNetwork pods have the same IP addresses as the nodes they reside on, their connections will be treated as node connections. For example, you can allow traffic from a hostNetwork Pod using an ipBlock rule.What you can't do with network policies (at least, not yet)As of Kubernetes 1.34, the following functionality does not exist in the NetworkPolicy API, but you might be able to implement workarounds using Operating System components (such as SELinux, OpenVSwitch, IPTables, and so on) or Layer 7 technologies (Ingress controllers, Service Mesh implementations) or admission controllers. In case you are new to network security in Kubernetes, its worth noting that the following User Stories cannot (yet) be implemented using the NetworkPolicy API.Forcing internal cluster traffic to go through a common gateway (this might be best served with a service mesh or other proxy).Anything TLS related (use a service mesh or ingress controller for this).Node specific policies (you can use CIDR notation for these, but you cannot target nodes by their Kubernetes identities specifically).Targeting of services by name (you can, however, target pods or namespaces by their labels, which is often a viable workaround).Creation or management of "Policy requests" that are fulfilled by a third party.Default policies which are applied to all namespaces or pods (there are some third party Kubernetes distributions and projects which can do this).Advanced policy querying and reachability tooling.The ability to log network security events (for example connections that are blocked or accepted).The ability to explicitly deny policies (currently the model for NetworkPolicies are deny by default, with only the ability to add allow rules).The ability to prevent loopback or incoming host traffic (Pods cannot currently block localhost access, nor do they have the ability to block access from their resident node).NetworkPolicy's impact on existing connectionsWhen the set of NetworkPolicies that applies to an existing connection changes - this could happen either due to a change in NetworkPolicies or if the relevant labels of the namespaces/pods selected by the policy (both subject and peers) are changed in the middle of an existing connection - it is implementation defined as to whether the change will take effect for that existing connection or not. Example: A policy is created that leads to denying a previously allowed connection, the underlying network plugin implementation is responsible for defining if that new policy will close the existing connections or not. It is recommended not to modify policies/pods/namespaces in ways that might affect existing connections.What's nextSee the Declare Network Policy walkthrough for further examples.See more recipes for common scenarios enabled by the NetworkPolicy resource.


# Using Node Authorization

Using Node AuthorizationNode authorization is a special-purpose authorization mode that specifically authorizes API requests made by kubelets.OverviewThe Node authorizer allows a kubelet to perform API operations. This includes:Read operations:servicesendpointsnodespodssecrets, configmaps, persistent volume claims and persistent volumes related to pods bound to the kubelet's nodeFEATURE STATE: Kubernetes v1.34 [stable] (enabled by default: true)Kubelets are limited to reading their own Node objects, and only reading pods bound to their node.Write operations:nodes and node status (enable the NodeRestriction admission plugin to limit a kubelet to modify its own node)pods and pod status (enable the NodeRestriction admission plugin to limit a kubelet to modify pods bound to itself)eventsAuth-related operations:read/write access to the CertificateSigningRequests API for TLS bootstrappingthe ability to create TokenReviews and SubjectAccessReviews for delegated authentication/authorization checksIn future releases, the node authorizer may add or remove permissions to ensure kubelets have the minimal set of permissions required to operate correctly.In order to be authorized by the Node authorizer, kubelets must use a credential that identifies them as being in the system:nodes group, with a username of system:node:<nodeName>. This group and user name format match the identity created for each kubelet as part of kubelet TLS bootstrapping.The value of <nodeName> must match precisely the name of the node as registered by the kubelet. By default, this is the host name as provided by hostname, or overridden via the kubelet option --hostname-override. However, when using the --cloud-provider kubelet option, the specific hostname may be determined by the cloud provider, ignoring the local hostname and the --hostname-override option. For specifics about how the kubelet determines the hostname, see the kubelet options reference.To enable the Node authorizer, start the API server with the --authorization-config flag set to a file that includes the Node authorizer; for example:apiVersion: apiserver.config.k8s.io/v1 kind: AuthorizationConfiguration authorizers: ... - type: Node ... Or, start the API server with the --authorization-mode flag set to a comma-separated list that includes Node; for example:kube-apiserver --authorization-mode=...,Node --other-options --more-options To limit the API objects kubelets are able to write, enable the NodeRestriction admission plugin by starting the apiserver with --enable-admission-plugins=...,NodeRestriction,...Migration considerationsKubelets outside the system:nodes groupKubelets outside the system:nodes group would not be authorized by the Node authorization mode, and would need to continue to be authorized via whatever mechanism currently authorizes them. The node admission plugin would not restrict requests from these kubelets.Kubelets with undifferentiated usernamesIn some deployments, kubelets have credentials that place them in the system:nodes group, but do not identify the particular node they are associated with, because they do not have a username in the system:node:... format. These kubelets would not be authorized by the Node authorization mode, and would need to continue to be authorized via whatever mechanism currently authorizes them.The NodeRestriction admission plugin would ignore requests from these kubelets, since the default node identifier implementation would not consider that a node identity.


# Generating Reference Documentation for the Kubernetes API

Generating Reference Documentation for the Kubernetes APIThis page shows how to update the Kubernetes API reference documentation.The Kubernetes API reference documentation is built from the Kubernetes OpenAPI spec using the kubernetes-sigs/reference-docs generation code.If you find bugs in the generated documentation, you need to fix them upstream.If you need only to regenerate the reference documentation from the OpenAPI spec, continue reading this page.Before you beginRequirements:You need a machine that is running Linux or macOS.You need to have these tools installed:Python v3.7.x+GitGolang version 1.13+Pip used to install PyYAMLPyYAML v5.1.2makegcc compiler/linkerDocker (Required only for kubectl command reference)Your PATH environment variable must include the required build tools, such as the Go binary and python.You need to know how to create a pull request to a GitHub repository. This involves creating your own fork of the repository. For more information, see Work from a local clone.Set up the local repositoriesCreate a local workspace and set your GOPATH:mkdir -p $HOME/<workspace> export GOPATH=$HOME/<workspace> Get a local clone of the following repositories:go get -u github.com/kubernetes-sigs/reference-docs go get -u github.com/go-openapi/loads go get -u github.com/go-openapi/spec If you don't already have the kubernetes/website repository, get it now:git clone https://github.com/<your-username>/website $GOPATH/src/github.com/<your-username>/website Get a clone of the kubernetes/kubernetes repository as k8s.io/kubernetes:git clone https://github.com/kubernetes/kubernetes $GOPATH/src/k8s.io/kubernetes The base directory of your clone of the kubernetes/kubernetes repository is $GOPATH/src/k8s.io/kubernetes. The remaining steps refer to your base directory as <k8s-base>.The base directory of your clone of the kubernetes/website repository is $GOPATH/src/github.com/<your username>/website. The remaining steps refer to your base directory as <web-base>.The base directory of your clone of the kubernetes-sigs/reference-docs repository is $GOPATH/src/github.com/kubernetes-sigs/reference-docs. The remaining steps refer to your base directory as <rdocs-base>.Generate the API reference docsThis section shows how to generate the published Kubernetes API reference documentation.Set build variablesSet K8S_ROOT to <k8s-base>.Set K8S_WEBROOT to <web-base>.Set K8S_RELEASE to the version of the docs you want to build. For example, if you want to build docs for Kubernetes 1.17.0, set K8S_RELEASE to 1.17.0.For example:export K8S_WEBROOT=${GOPATH}/src/github.com/<your-username>/website export K8S_ROOT=${GOPATH}/src/k8s.io/kubernetes export K8S_RELEASE=1.17.0 Create versioned directory and fetch Open API specThe updateapispec build target creates the versioned build directory. After the directory is created, the Open API spec is fetched from the <k8s-base> repository. These steps ensure that the version of the configuration files and Kubernetes Open API spec match the release version. The versioned directory name follows the pattern of v<major>_<minor>.In the <rdocs-base> directory, run the following build target:cd <rdocs-base> make updateapispec Build the API reference docsThe copyapi target builds the API reference and copies the generated files to directories in <web-base>. Run the following command in <rdocs-base>:cd <rdocs-base> make copyapi Verify that these two files have been generated:[ -e "<rdocs-base>/gen-apidocs/build/index.html" ] && echo "index.html built" || echo "no index.html" [ -e "<rdocs-base>/gen-apidocs/build/navData.js" ] && echo "navData.js built" || echo "no navData.js" Go to the base of your local <web-base>, and view which files have been modified:cd <web-base> git status The output is similar to:static/docs/reference/generated/kubernetes-api/v1.34/css/bootstrap.min.css static/docs/reference/generated/kubernetes-api/v1.34/css/font-awesome.min.css static/docs/reference/generated/kubernetes-api/v1.34/css/stylesheet.css static/docs/reference/generated/kubernetes-api/v1.34/fonts/FontAwesome.otf static/docs/reference/generated/kubernetes-api/v1.34/fonts/fontawesome-webfont.eot static/docs/reference/generated/kubernetes-api/v1.34/fonts/fontawesome-webfont.svg static/docs/reference/generated/kubernetes-api/v1.34/fonts/fontawesome-webfont.ttf static/docs/reference/generated/kubernetes-api/v1.34/fonts/fontawesome-webfont.woff static/docs/reference/generated/kubernetes-api/v1.34/fonts/fontawesome-webfont.woff2 static/docs/reference/generated/kubernetes-api/v1.34/index.html static/docs/reference/generated/kubernetes-api/v1.34/js/jquery.scrollTo.min.js static/docs/reference/generated/kubernetes-api/v1.34/js/navData.js static/docs/reference/generated/kubernetes-api/v1.34/js/scroll.js Update the API reference index pagesWhen generating reference documentation for a new release, update the file, <web-base>/content/en/docs/reference/kubernetes-api/api-index.md with the new version number.Open <web-base>/content/en/docs/reference/kubernetes-api/api-index.md for editing, and update the API reference version number. For example:--- title: v1.17 --- [Kubernetes API v1.17](/docs/reference/generated/kubernetes-api/v1.17/) Open <web-base>/content/en/docs/reference/_index.md for editing, and add a new link for the latest API reference. Remove the oldest API reference version. There should be five links to the most recent API references.Locally test the API referencePublish a local version of the API reference. Verify the local preview.cd <web-base> git submodule update --init --recursive --depth 1 # if not already done make container-serve Commit the changesIn <web-base>, run git add and git commit to commit the change.Submit your changes as a pull request to the kubernetes/website repository. Monitor your pull request, and respond to reviewer comments as needed. Continue to monitor your pull request until it has been merged.What's nextGenerating Reference Documentation QuickstartGenerating Reference Docs for Kubernetes Components and ToolsGenerating Reference Documentation for kubectl Commands


# Protocols for Services

Protocols for ServicesIf you configure a Service, you can select from any network protocol that Kubernetes supports.Kubernetes supports the following protocols with Services:SCTPTCP (the default)UDPWhen you define a Service, you can also specify the application protocol that it uses.This document details some special cases, all of them typically using TCP as a transport protocol:HTTP and HTTPSPROXY protocolTLS termination at the load balancerSupported protocolsThere are 3 valid values for the protocol of a port for a Service:SCTPFEATURE STATE: Kubernetes v1.20 [stable]When using a network plugin that supports SCTP traffic, you can use SCTP for most Services. For type: LoadBalancer Services, SCTP support depends on the cloud provider offering this facility. (Most do not).SCTP is not supported on nodes that run Windows.Support for multihomed SCTP associationsThe support of multihomed SCTP associations requires that the CNI plugin can support the assignment of multiple interfaces and IP addresses to a Pod.NAT for multihomed SCTP associations requires special logic in the corresponding kernel modules.TCPYou can use TCP for any kind of Service, and it's the default network protocol.UDPYou can use UDP for most Services. For type: LoadBalancer Services, UDP support depends on the cloud provider offering this facility.Special casesHTTPIf your cloud provider supports it, you can use a Service in LoadBalancer mode to configure a load balancer outside of your Kubernetes cluster, in a special mode where your cloud provider's load balancer implements HTTP / HTTPS reverse proxying, with traffic forwarded to the backend endpoints for that Service.Typically, you set the protocol for the Service to TCP and add an annotation (usually specific to your cloud provider) that configures the load balancer to handle traffic at the HTTP level. This configuration might also include serving HTTPS (HTTP over TLS) and reverse-proxying plain HTTP to your workload.Note:You can also use an Ingress to expose HTTP/HTTPS Services.You might additionally want to specify that the application protocol of the connection is http or https. Use http if the session from the load balancer to your workload is HTTP without TLS, and use https if the session from the load balancer to your workload uses TLS encryption.PROXY protocolIf your cloud provider supports it, you can use a Service set to type: LoadBalancer to configure a load balancer outside of Kubernetes itself, that will forward connections wrapped with the PROXY protocol.The load balancer then sends an initial series of octets describing the incoming connection, similar to this example (PROXY protocol v1):PROXY TCP4 192.0.2.202 10.0.42.7 12345 7\r\n The data after the proxy protocol preamble are the original data from the client. When either side closes the connection, the load balancer also triggers a connection close and sends any remaining data where feasible.Typically, you define a Service with the protocol to TCP. You also set an annotation, specific to your cloud provider, that configures the load balancer to wrap each incoming connection in the PROXY protocol.TLSIf your cloud provider supports it, you can use a Service set to type: LoadBalancer as a way to set up external reverse proxying, where the connection from client to load balancer is TLS encrypted and the load balancer is the TLS server peer. The connection from the load balancer to your workload can also be TLS, or might be plain text. The exact options available to you depend on your cloud provider or custom Service implementation.Typically, you set the protocol to TCP and set an annotation (usually specific to your cloud provider) that configures the load balancer to act as a TLS server. You would configure the TLS identity (as server, and possibly also as a client that connects to your workload) using mechanisms that are specific to your cloud provider.


# Cloud Native Security and Kubernetes

Cloud Native Security and KubernetesConcepts for keeping your cloud-native workload secure.Kubernetes is based on a cloud-native architecture, and draws on advice from the CNCF about good practice for cloud native information security.Read on through this page for an overview of how Kubernetes is designed to help you deploy a secure cloud native platform.Cloud native information securityThe CNCF white paper on cloud native security defines security controls and practices that are appropriate to different lifecycle phases.Develop lifecycle phaseEnsure the integrity of development environments.Design applications following good practice for information security, appropriate for your context.Consider end user security as part of solution design.To achieve this, you can:Adopt an architecture, such as zero trust, that minimizes attack surfaces, even for internal threats.Define a code review process that considers security concerns.Build a threat model of your system or application that identifies trust boundaries. Use that to model to identify risks and to help find ways to treat those risks.Incorporate advanced security automation, such as fuzzing and security chaos engineering, where it's justified.Distribute lifecycle phaseEnsure the security of the supply chain for container images you execute.Ensure the security of the supply chain for the cluster and other components that execute your application. An example of another component might be an external database that your cloud-native application uses for persistence.To achieve this, you can:Scan container images and other artifacts for known vulnerabilities.Ensure that software distribution uses encryption in transit, with a chain of trust for the software source.Adopt and follow processes to update dependencies when updates are available, especially in response to security announcements.Use validation mechanisms such as digital certificates for supply chain assurance.Subscribe to feeds and other mechanisms to alert you to security risks.Restrict access to artifacts. Place container images in a private registry that only allows authorized clients to pull images.Deploy lifecycle phaseEnsure appropriate restrictions on what can be deployed, who can deploy it, and where it can be deployed to. You can enforce measures from the distribute phase, such as verifying the cryptographic identity of container image artifacts.You can deploy different applications and cluster components into different namespaces. Containers themselves, and namespaces, both provide isolation mechanisms that are relevant to information security.When you deploy Kubernetes, you also set the foundation for your applications' runtime environment: a Kubernetes cluster (or multiple clusters). That IT infrastructure must provide the security guarantees that higher layers expect.Runtime lifecycle phaseThe Runtime phase comprises three critical areas: access, compute, and storage.Runtime protection: accessThe Kubernetes API is what makes your cluster work. Protecting this API is key to providing effective cluster security.Other pages in the Kubernetes documentation have more detail about how to set up specific aspects of access control. The security checklist has a set of suggested basic checks for your cluster.Beyond that, securing your cluster means implementing effective authentication and authorization for API access. Use ServiceAccounts to provide and manage security identities for workloads and cluster components.Kubernetes uses TLS to protect API traffic; make sure to deploy the cluster using TLS (including for traffic between nodes and the control plane), and protect the encryption keys. If you use Kubernetes' own API for CertificateSigningRequests, pay special attention to restricting misuse there.Runtime protection: computeContainers provide two things: isolation between different applications, and a mechanism to combine those isolated applications to run on the same host computer. Those two aspects, isolation and aggregation, mean that runtime security involves identifying trade-offs and finding an appropriate balance.Kubernetes relies on a container runtime to actually set up and run containers. The Kubernetes project does not recommend a specific container runtime and you should make sure that the runtime(s) that you choose meet your information security needs.To protect your compute at runtime, you can:Enforce Pod security standards for applications, to help ensure they run with only the necessary privileges.Run a specialized operating system on your nodes that is designed specifically for running containerized workloads. This is typically based on a read-only operating system (immutable image) that provides only the services essential for running containers.Container-specific operating systems help to isolate system components and present a reduced attack surface in case of a container escape.Define ResourceQuotas to fairly allocate shared resources, and use mechanisms such as LimitRanges to ensure that Pods specify their resource requirements.Partition workloads across different nodes. Use node isolation mechanisms, either from Kubernetes itself or from the ecosystem, to ensure that Pods with different trust contexts are run on separate sets of nodes.Use a container runtime that provides security restrictions.On Linux nodes, use a Linux security module such as AppArmor or seccomp.Runtime protection: storageTo protect storage for your cluster and the applications that run there, you can:Integrate your cluster with an external storage plugin that provides encryption at rest for volumes.Enable encryption at rest for API objects.Protect data durability using backups. Verify that you can restore these, whenever you need to.Authenticate connections between cluster nodes and any network storage they rely upon.Implement data encryption within your own application.For encryption keys, generating these within specialized hardware provides the best protection against disclosure risks. A hardware security module can let you perform cryptographic operations without allowing the security key to be copied elsewhere.Networking and securityYou should also consider network security measures, such as NetworkPolicy or a service mesh. Some network plugins for Kubernetes provide encryption for your cluster network, using technologies such as a virtual private network (VPN) overlay. By design, Kubernetes lets you use your own networking plugin for your cluster (if you use managed Kubernetes, the person or organization managing your cluster may have chosen a network plugin for you).The network plugin you choose and the way you integrate it can have a strong impact on the security of information in transit.Observability and runtime securityKubernetes lets you extend your cluster with extra tooling. You can set up third party solutions to help you monitor or troubleshoot your applications and the clusters they are running. You also get some basic observability features built in to Kubernetes itself. Your code running in containers can generate logs, publish metrics or provide other observability data; at deploy time, you need to make sure your cluster provides an appropriate level of protection there.If you set up a metrics dashboard or something similar, review the chain of components that populate data into that dashboard, as well as the dashboard itself. Make sure that the whole chain is designed with enough resilience and enough integrity protection that you can rely on it even during an incident where your cluster might be degraded.Where appropriate, deploy security measures below the level of Kubernetes itself, such as cryptographically measured boot, or authenticated distribution of time (which helps ensure the fidelity of logs and audit records).For a high assurance environment, deploy cryptographic protections to ensure that logs are both tamper-proof and confidential.What's nextCloud native securityCNCF white paper on cloud native security.CNCF white paper on good practices for securing a software supply chain.Fixing the Kubernetes clusterf**k: Understanding security from the kernel up (FOSDEM 2020)Kubernetes Security Best Practices (Kubernetes Forum Seoul 2019)Towards Measured Boot Out of the Box (Linux Security Summit 2016)Kubernetes and information securityKubernetes securitySecuring your clusterData encryption in transit for the control planeData encryption at restSecrets in KubernetesControlling Access to the Kubernetes APINetwork policies for PodsPod security standardsRuntimeClasses


# Deprecated API Migration Guide

Deprecated API Migration GuideAs the Kubernetes API evolves, APIs are periodically reorganized or upgraded. When APIs evolve, the old API is deprecated and eventually removed. This page contains information you need to know when migrating from deprecated API versions to newer and more stable API versions.Removed APIs by releasev1.32The v1.32 release stopped serving the following deprecated API versions:Flow control resourcesThe flowcontrol.apiserver.k8s.io/v1beta3 API version of FlowSchema and PriorityLevelConfiguration is no longer served as of v1.32.Migrate manifests and API clients to use the flowcontrol.apiserver.k8s.io/v1 API version, available since v1.29.All existing persisted objects are accessible via the new APINotable changes in flowcontrol.apiserver.k8s.io/v1:The PriorityLevelConfiguration spec.limited.nominalConcurrencyShares field only defaults to 30 when unspecified, and an explicit value of 0 is not changed to 30.v1.29The v1.29 release stopped serving the following deprecated API versions:Flow control resourcesThe flowcontrol.apiserver.k8s.io/v1beta2 API version of FlowSchema and PriorityLevelConfiguration is no longer served as of v1.29.Migrate manifests and API clients to use the flowcontrol.apiserver.k8s.io/v1 API version, available since v1.29, or the flowcontrol.apiserver.k8s.io/v1beta3 API version, available since v1.26.All existing persisted objects are accessible via the new APINotable changes in flowcontrol.apiserver.k8s.io/v1:The PriorityLevelConfiguration spec.limited.assuredConcurrencyShares field is renamed to spec.limited.nominalConcurrencyShares and only defaults to 30 when unspecified, and an explicit value of 0 is not changed to 30.Notable changes in flowcontrol.apiserver.k8s.io/v1beta3:The PriorityLevelConfiguration spec.limited.assuredConcurrencyShares field is renamed to spec.limited.nominalConcurrencySharesv1.27The v1.27 release stopped serving the following deprecated API versions:CSIStorageCapacityThe storage.k8s.io/v1beta1 API version of CSIStorageCapacity is no longer served as of v1.27.Migrate manifests and API clients to use the storage.k8s.io/v1 API version, available since v1.24.All existing persisted objects are accessible via the new APINo notable changesv1.26The v1.26 release stopped serving the following deprecated API versions:Flow control resourcesThe flowcontrol.apiserver.k8s.io/v1beta1 API version of FlowSchema and PriorityLevelConfiguration is no longer served as of v1.26.Migrate manifests and API clients to use the flowcontrol.apiserver.k8s.io/v1beta2 API version.All existing persisted objects are accessible via the new APINo notable changesHorizontalPodAutoscalerThe autoscaling/v2beta2 API version of HorizontalPodAutoscaler is no longer served as of v1.26.Migrate manifests and API clients to use the autoscaling/v2 API version, available since v1.23.All existing persisted objects are accessible via the new APINotable changes:targetAverageUtilization is replaced with target.averageUtilization and target.type: Utilization. See Autoscaling on multiple metrics and custom metrics.v1.25The v1.25 release stopped serving the following deprecated API versions:CronJobThe batch/v1beta1 API version of CronJob is no longer served as of v1.25.Migrate manifests and API clients to use the batch/v1 API version, available since v1.21.All existing persisted objects are accessible via the new APINo notable changesEndpointSliceThe discovery.k8s.io/v1beta1 API version of EndpointSlice is no longer served as of v1.25.Migrate manifests and API clients to use the discovery.k8s.io/v1 API version, available since v1.21.All existing persisted objects are accessible via the new APINotable changes in discovery.k8s.io/v1:use per Endpoint nodeName field instead of deprecated topology["kubernetes.io/hostname"] fielduse per Endpoint zone field instead of deprecated topology["topology.kubernetes.io/zone"] fieldtopology is replaced with the deprecatedTopology field which is not writable in v1EventThe events.k8s.io/v1beta1 API version of Event is no longer served as of v1.25.Migrate manifests and API clients to use the events.k8s.io/v1 API version, available since v1.19.All existing persisted objects are accessible via the new APINotable changes in events.k8s.io/v1:type is limited to Normal and WarninginvolvedObject is renamed to regardingaction, reason, reportingController, and reportingInstance are required when creating new events.k8s.io/v1 Eventsuse eventTime instead of the deprecated firstTimestamp field (which is renamed to deprecatedFirstTimestamp and not permitted in new events.k8s.io/v1 Events)use series.lastObservedTime instead of the deprecated lastTimestamp field (which is renamed to deprecatedLastTimestamp and not permitted in new events.k8s.io/v1 Events)use series.count instead of the deprecated count field (which is renamed to deprecatedCount and not permitted in new events.k8s.io/v1 Events)use reportingController instead of the deprecated source.component field (which is renamed to deprecatedSource.component and not permitted in new events.k8s.io/v1 Events)use reportingInstance instead of the deprecated source.host field (which is renamed to deprecatedSource.host and not permitted in new events.k8s.io/v1 Events)HorizontalPodAutoscalerThe autoscaling/v2beta1 API version of HorizontalPodAutoscaler is no longer served as of v1.25.Migrate manifests and API clients to use the autoscaling/v2 API version, available since v1.23.All existing persisted objects are accessible via the new APINotable changes:targetAverageUtilization is replaced with target.averageUtilization and target.type: Utilization. See Autoscaling on multiple metrics and custom metrics.PodDisruptionBudgetThe policy/v1beta1 API version of PodDisruptionBudget is no longer served as of v1.25.Migrate manifests and API clients to use the policy/v1 API version, available since v1.21.All existing persisted objects are accessible via the new APINotable changes in policy/v1:an empty spec.selector ({}) written to a policy/v1 PodDisruptionBudget selects all pods in the namespace (in policy/v1beta1 an empty spec.selector selected no pods). An unset spec.selector selects no pods in either API version.PodSecurityPolicyPodSecurityPolicy in the policy/v1beta1 API version is no longer served as of v1.25, and the PodSecurityPolicy admission controller will be removed.Migrate to Pod Security Admission or a 3rd party admission webhook. For a migration guide, see Migrate from PodSecurityPolicy to the Built-In PodSecurity Admission Controller. For more information on the deprecation, see PodSecurityPolicy Deprecation: Past, Present, and Future.RuntimeClassRuntimeClass in the node.k8s.io/v1beta1 API version is no longer served as of v1.25.Migrate manifests and API clients to use the node.k8s.io/v1 API version, available since v1.20.All existing persisted objects are accessible via the new APINo notable changesv1.22The v1.22 release stopped serving the following deprecated API versions:Webhook resourcesThe admissionregistration.k8s.io/v1beta1 API version of MutatingWebhookConfiguration and ValidatingWebhookConfiguration is no longer served as of v1.22.Migrate manifests and API clients to use the admissionregistration.k8s.io/v1 API version, available since v1.16.All existing persisted objects are accessible via the new APIsNotable changes:webhooks[*].failurePolicy default changed from Ignore to Fail for v1webhooks[*].matchPolicy default changed from Exact to Equivalent for v1webhooks[*].timeoutSeconds default changed from 30s to 10s for v1webhooks[*].sideEffects default value is removed, and the field made required, and only None and NoneOnDryRun are permitted for v1webhooks[*].admissionReviewVersions default value is removed and the field made required for v1 (supported versions for AdmissionReview are v1 and v1beta1)webhooks[*].name must be unique in the list for objects created via admissionregistration.k8s.io/v1CustomResourceDefinitionThe apiextensions.k8s.io/v1beta1 API version of CustomResourceDefinition is no longer served as of v1.22.Migrate manifests and API clients to use the apiextensions.k8s.io/v1 API version, available since v1.16.All existing persisted objects are accessible via the new APINotable changes:spec.scope is no longer defaulted to Namespaced and must be explicitly specifiedspec.version is removed in v1; use spec.versions insteadspec.validation is removed in v1; use spec.versions[*].schema insteadspec.subresources is removed in v1; use spec.versions[*].subresources insteadspec.additionalPrinterColumns is removed in v1; use spec.versions[*].additionalPrinterColumns insteadspec.conversion.webhookClientConfig is moved to spec.conversion.webhook.clientConfig in v1spec.conversion.conversionReviewVersions is moved to spec.conversion.webhook.conversionReviewVersions in v1spec.versions[*].schema.openAPIV3Schema is now required when creating v1 CustomResourceDefinition objects, and must be a structural schemaspec.preserveUnknownFields: true is disallowed when creating v1 CustomResourceDefinition objects; it must be specified within schema definitions as x-kubernetes-preserve-unknown-fields: trueIn additionalPrinterColumns items, the JSONPath field was renamed to jsonPath in v1 (fixes #66531)APIServiceThe apiregistration.k8s.io/v1beta1 API version of APIService is no longer served as of v1.22.Migrate manifests and API clients to use the apiregistration.k8s.io/v1 API version, available since v1.10.All existing persisted objects are accessible via the new APINo notable changesTokenReviewThe authentication.k8s.io/v1beta1 API version of TokenReview is no longer served as of v1.22.Migrate manifests and API clients to use the authentication.k8s.io/v1 API version, available since v1.6.No notable changesSubjectAccessReview resourcesThe authorization.k8s.io/v1beta1 API version of LocalSubjectAccessReview, SelfSubjectAccessReview, SubjectAccessReview, and SelfSubjectRulesReview is no longer served as of v1.22.Migrate manifests and API clients to use the authorization.k8s.io/v1 API version, available since v1.6.Notable changes:spec.group was renamed to spec.groups in v1 (fixes #32709)CertificateSigningRequestThe certificates.k8s.io/v1beta1 API version of CertificateSigningRequest is no longer served as of v1.22.Migrate manifests and API clients to use the certificates.k8s.io/v1 API version, available since v1.19.All existing persisted objects are accessible via the new APINotable changes in certificates.k8s.io/v1:For API clients requesting certificates:spec.signerName is now required (see known Kubernetes signers), and requests for kubernetes.io/legacy-unknown are not allowed to be created via the certificates.k8s.io/v1 APIspec.usages is now required, may not contain duplicate values, and must only contain known usagesFor API clients approving or signing certificates:status.conditions may not contain duplicate typesstatus.conditions[*].status is now requiredstatus.certificate must be PEM-encoded, and contain only CERTIFICATE blocksLeaseThe coordination.k8s.io/v1beta1 API version of Lease is no longer served as of v1.22.Migrate manifests and API clients to use the coordination.k8s.io/v1 API version, available since v1.14.All existing persisted objects are accessible via the new APINo notable changesIngressThe extensions/v1beta1 and networking.k8s.io/v1beta1 API versions of Ingress is no longer served as of v1.22.Migrate manifests and API clients to use the networking.k8s.io/v1 API version, available since v1.19.All existing persisted objects are accessible via the new APINotable changes:spec.backend is renamed to spec.defaultBackendThe backend serviceName field is renamed to service.nameNumeric backend servicePort fields are renamed to service.port.numberString backend servicePort fields are renamed to service.port.namepathType is now required for each specified path. Options are Prefix, Exact, and ImplementationSpecific. To match the undefined v1beta1 behavior, use ImplementationSpecific.IngressClassThe networking.k8s.io/v1beta1 API version of IngressClass is no longer served as of v1.22.Migrate manifests and API clients to use the networking.k8s.io/v1 API version, available since v1.19.All existing persisted objects are accessible via the new APINo notable changesRBAC resourcesThe rbac.authorization.k8s.io/v1beta1 API version of ClusterRole, ClusterRoleBinding, Role, and RoleBinding is no longer served as of v1.22.Migrate manifests and API clients to use the rbac.authorization.k8s.io/v1 API version, available since v1.8.All existing persisted objects are accessible via the new APIsNo notable changesPriorityClassThe scheduling.k8s.io/v1beta1 API version of PriorityClass is no longer served as of v1.22.Migrate manifests and API clients to use the scheduling.k8s.io/v1 API version, available since v1.14.All existing persisted objects are accessible via the new APINo notable changesStorage resourcesThe storage.k8s.io/v1beta1 API version of CSIDriver, CSINode, StorageClass, and VolumeAttachment is no longer served as of v1.22.Migrate manifests and API clients to use the storage.k8s.io/v1 API versionCSIDriver is available in storage.k8s.io/v1 since v1.19.CSINode is available in storage.k8s.io/v1 since v1.17StorageClass is available in storage.k8s.io/v1 since v1.6VolumeAttachment is available in storage.k8s.io/v1 v1.13All existing persisted objects are accessible via the new APIsNo notable changesv1.16The v1.16 release stopped serving the following deprecated API versions:NetworkPolicyThe extensions/v1beta1 API version of NetworkPolicy is no longer served as of v1.16.Migrate manifests and API clients to use the networking.k8s.io/v1 API version, available since v1.8.All existing persisted objects are accessible via the new APIDaemonSetThe extensions/v1beta1 and apps/v1beta2 API versions of DaemonSet are no longer served as of v1.16.Migrate manifests and API clients to use the apps/v1 API version, available since v1.9.All existing persisted objects are accessible via the new APINotable changes:spec.templateGeneration is removedspec.selector is now required and immutable after creation; use the existing template labels as the selector for seamless upgradesspec.updateStrategy.type now defaults to RollingUpdate (the default in extensions/v1beta1 was OnDelete)DeploymentThe extensions/v1beta1, apps/v1beta1, and apps/v1beta2 API versions of Deployment are no longer served as of v1.16.Migrate manifests and API clients to use the apps/v1 API version, available since v1.9.All existing persisted objects are accessible via the new APINotable changes:spec.rollbackTo is removedspec.selector is now required and immutable after creation; use the existing template labels as the selector for seamless upgradesspec.progressDeadlineSeconds now defaults to 600 seconds (the default in extensions/v1beta1 was no deadline)spec.revisionHistoryLimit now defaults to 10 (the default in apps/v1beta1 was 2, the default in extensions/v1beta1 was to retain all)maxSurge and maxUnavailable now default to 25% (the default in extensions/v1beta1 was 1)StatefulSetThe apps/v1beta1 and apps/v1beta2 API versions of StatefulSet are no longer served as of v1.16.Migrate manifests and API clients to use the apps/v1 API version, available since v1.9.All existing persisted objects are accessible via the new APINotable changes:spec.selector is now required and immutable after creation; use the existing template labels as the selector for seamless upgradesspec.updateStrategy.type now defaults to RollingUpdate (the default in apps/v1beta1 was OnDelete)ReplicaSetThe extensions/v1beta1, apps/v1beta1, and apps/v1beta2 API versions of ReplicaSet are no longer served as of v1.16.Migrate manifests and API clients to use the apps/v1 API version, available since v1.9.All existing persisted objects are accessible via the new APINotable changes:spec.selector is now required and immutable after creation; use the existing template labels as the selector for seamless upgradesPodSecurityPolicyThe extensions/v1beta1 API version of PodSecurityPolicy is no longer served as of v1.16.Migrate manifests and API client to use the policy/v1beta1 API version, available since v1.10.Note that the policy/v1beta1 API version of PodSecurityPolicy will be removed in v1.25.What to doTest with deprecated APIs disabledYou can test your clusters by starting an API server with specific API versions disabled to simulate upcoming removals. Add the following flag to the API server startup arguments:--runtime-config=<group>/<version>=falseFor example:--runtime-config=admissionregistration.k8s.io/v1beta1=false,apiextensions.k8s.io/v1beta1,...Locate use of deprecated APIsUse client warnings, metrics, and audit information available in 1.19+ to locate use of deprecated APIs.Migrate to non-deprecated APIsUpdate custom integrations and controllers to call the non-deprecated APIsChange YAML files to reference the non-deprecated APIsYou can use the kubectl convert command to automatically convert an existing object:kubectl convert -f <file> --output-version <group>/<version>.For example, to convert an older Deployment to apps/v1, you can run:kubectl convert -f ./my-deployment.yaml --output-version apps/v1This conversion may use non-ideal default values. To learn more about a specific resource, check the Kubernetes API reference.Note:The kubectl convert tool is not installed by default, although in fact it once was part of kubectl itself. For more details, you can read the deprecation and removal issue for the built-in subcommand.To learn how to set up kubectl convert on your computer, visit the page that is right for your operating system: Linux, macOS, or Windows.


# Articles on dockershim Removal and on Using CRI-compatible Runtimes

Articles on dockershim Removal and on Using CRI-compatible RuntimesThis is a list of articles and other pages that are either about the Kubernetes' deprecation and removal of dockershim, or about using CRI-compatible container runtimes, in connection with that removal.Kubernetes projectKubernetes blog: Dockershim Removal FAQ (originally published 2020/12/02)Kubernetes blog: Updated: Dockershim Removal FAQ (updated published 2022/02/17)Kubernetes blog: Kubernetes is Moving on From Dockershim: Commitments and Next Steps (published 2022/01/07)Kubernetes blog: Dockershim removal is coming. Are you ready? (published 2021/11/12)Kubernetes documentation: Migrating from dockershimKubernetes documentation: Container RuntimesKubernetes enhancement proposal: KEP-2221: Removing dockershim from kubeletKubernetes enhancement proposal issue: Removing dockershim from kubelet (k/enhancements#2221)You can provide


# kubectl auth reconcile

kubectl auth reconcileSynopsisReconciles rules for RBAC role, role binding, cluster role, and cluster role binding objects.Missing objects are created, and the containing namespace is created for namespaced objects, if required.Existing roles are updated to include the permissions in the input objects, and remove extra permissions if --remove-extra-permissions is specified.Existing bindings are updated to include the subjects in the input objects, and remove extra subjects if --remove-extra-subjects is specified.This is preferred to 'apply' for RBAC resources so that semantically-aware merging of rules and subjects is done.kubectl auth reconcile -f FILENAME Examples # Reconcile RBAC resources from a file kubectl auth reconcile -f my-rbac-rules.yaml Options--allow-missing-template-keys Default: trueIf true, ignore any errors in templates when a field or map key is missing in the template. Only applies to golang and jsonpath output formats.--dry-run string[="unchanged"] Default: "none"Must be "none", "server", or "client". If client strategy, only print the object that would be sent, without sending it. If server strategy, submit server-side request without persisting the resource.-f, --filename stringsFilename, directory, or URL to files identifying the resource to reconcile.-h, --helphelp for reconcile-k, --kustomize stringProcess the kustomization directory. This flag can't be used together with -f or -R.-o, --output stringOutput format. One of: (json, yaml, name, go-template, go-template-file, template, templatefile, jsonpath, jsonpath-as-json, jsonpath-file).-R, --recursiveProcess the directory used in -f, --filename recursively. Useful when you want to manage related manifests organized within the same directory.--remove-extra-permissionsIf true, removes extra permissions added to roles--remove-extra-subjectsIf true, removes extra subjects added to rolebindings--show-managed-fieldsIf true, keep the managedFields when printing objects in JSON or YAML format.--template stringTemplate string or path to template file to use when -o=go-template, -o=go-template-file. The template format is golang templates [http://golang.org/pkg/text/template/#pkg-overview].Parent Options Inherited--as stringUsername to impersonate for the operation. User could be a regular user or a service account in a namespace.--as-group stringsGroup to impersonate for the operation, this flag can be repeated to specify multiple groups.--as-uid stringUID to impersonate for the operation.--cache-dir string Default: "$HOME/.kube/cache"Default cache directory--certificate-authority stringPath to a cert file for the certificate authority--client-certificate stringPath to a client certificate file for TLS--client-key stringPath to a client key file for TLS--cluster stringThe name of the kubeconfig cluster to use--context stringThe name of the kubeconfig context to use--disable-compressionIf true, opt-out of response compression for all requests to the server--insecure-skip-tls-verifyIf true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure--kubeconfig stringPath to the kubeconfig file to use for CLI requests.--kuberc stringPath to the kuberc file to use for preferences. This can be disabled by exporting KUBECTL_KUBERC=false feature gate or turning off the feature KUBERC=off.--match-server-versionRequire server version to match client version-n, --namespace stringIf present, the namespace scope for this CLI request--password stringPassword for basic authentication to the API server--profile string Default: "none"Name of profile to capture. One of (none|cpu|heap|goroutine|threadcreate|block|mutex)--profile-output string Default: "profile.pprof"Name of the file to write the profile to--request-timeout string Default: "0"The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests.-s, --server stringThe address and port of the Kubernetes API server--storage-driver-buffer-duration duration Default: 1m0sWrites in the storage driver will be buffered for this duration, and committed to the non memory backends as a single transaction--storage-driver-db string Default: "cadvisor"database name--storage-driver-host string Default: "localhost:8086"database host:port--storage-driver-password string Default: "root"database password--storage-driver-secureuse secure connection with database--storage-driver-table string Default: "stats"table name--storage-driver-user string Default: "root"database username--tls-server-name stringServer name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used--token stringBearer token for authentication to the API server--user stringThe name of the kubeconfig user to use--username stringUsername for basic authentication to the API server--version version[=true]--version, --version=raw prints version information and quits; --version=vX.Y.Z... sets the reported version--warnings-as-errorsTreat warnings received from the server as errors and exit with a non-zero exit codeSee Alsokubectl auth - Inspect authorizationThis page is automatically generated.If you plan to report an issue with this page, mention that the page is auto-generated in your issue description. The fix may need to happen elsewhere in the Kubernetes project.


# Inject Data Into Applications

Inject Data Into ApplicationsSpecify configuration and other data for the Pods that run your workload.Define a Command and Arguments for a ContainerDefine Dependent Environment VariablesDefine Environment Variables for a ContainerDefine Environment Variable Values Using An Init ContainerExpose Pod Information to Containers Through Environment VariablesExpose Pod Information to Containers Through FilesDistribute Credentials Securely Using Secrets


# Configure RunAsUserName for Windows pods and containers

Configure RunAsUserName for Windows pods and containersFEATURE STATE: Kubernetes v1.18 [stable]This page shows how to use the runAsUserName setting for Pods and containers that will run on Windows nodes. This is roughly equivalent of the Linux-specific runAsUser setting, allowing you to run applications in a container as a different username than the default.Before you beginYou need to have a Kubernetes cluster and the kubectl command-line tool must be configured to communicate with your cluster. The cluster is expected to have Windows worker nodes where pods with containers running Windows workloads will get scheduled.Set the Username for a PodTo specify the username with which to execute the Pod's container processes, include the securityContext field (PodSecurityContext) in the Pod specification, and within it, the windowsOptions (WindowsSecurityContextOptions) field containing the runAsUserName field.The Windows security context options that you specify for a Pod apply to all Containers and init Containers in the Pod.Here is a configuration file for a Windows Pod that has the runAsUserName field set:windows/run-as-username-pod.yaml apiVersion: v1 kind: Pod metadata: name: run-as-username-pod-demo spec: securityContext: windowsOptions: runAsUserName: "ContainerUser" containers: - name: run-as-username-demo image: mcr.microsoft.com/windows/servercore:ltsc2019 command: ["ping", "-t", "localhost"] nodeSelector: kubernetes.io/os: windows Create the Pod:kubectl apply -f https://k8s.io/examples/windows/run-as-username-pod.yaml Verify that the Pod's Container is running:kubectl get pod run-as-username-pod-demo Get a shell to the running Container:kubectl exec -it run-as-username-pod-demo -- powershell Check that the shell is running user the correct username:echo $env:USERNAME The output should be:ContainerUser Set the Username for a ContainerTo specify the username with which to execute a Container's processes, include the securityContext field (SecurityContext) in the Container manifest, and within it, the windowsOptions (WindowsSecurityContextOptions) field containing the runAsUserName field.The Windows security context options that you specify for a Container apply only to that individual Container, and they override the settings made at the Pod level.Here is the configuration file for a Pod that has one Container, and the runAsUserName field is set at the Pod level and the Container level:windows/run-as-username-container.yaml apiVersion: v1 kind: Pod metadata: name: run-as-username-container-demo spec: securityContext: windowsOptions: runAsUserName: "ContainerUser" containers: - name: run-as-username-demo image: mcr.microsoft.com/windows/servercore:ltsc2019 command: ["ping", "-t", "localhost"] securityContext: windowsOptions: runAsUserName: "ContainerAdministrator" nodeSelector: kubernetes.io/os: windows Create the Pod:kubectl apply -f https://k8s.io/examples/windows/run-as-username-container.yaml Verify that the Pod's Container is running:kubectl get pod run-as-username-container-demo Get a shell to the running Container:kubectl exec -it run-as-username-container-demo -- powershell Check that the shell is running user the correct username (the one set at the Container level):echo $env:USERNAME The output should be:ContainerAdministrator Windows Username limitationsIn order to use this feature, the value set in the runAsUserName field must be a valid username. It must have the following format: DOMAIN\USER, where DOMAIN\ is optional. Windows user names are case insensitive. Additionally, there are some restrictions regarding the DOMAIN and USER:The runAsUserName field cannot be empty, and it cannot contain control characters (ASCII values: 0x00-0x1F, 0x7F)The DOMAIN must be either a NetBios name, or a DNS name, each with their own restrictions:NetBios names: maximum 15 characters, cannot start with . (dot), and cannot contain the following characters: \ / : * ? " < > |DNS names: maximum 255 characters, contains only alphanumeric characters, dots, and dashes, and it cannot start or end with a . (dot) or - (dash).The USER must have at most 20 characters, it cannot contain only dots or spaces, and it cannot contain the following characters: " / \ [ ] : ; | = , + * ? < > @.Examples of acceptable values for the runAsUserName field: ContainerAdministrator, ContainerUser, NT AUTHORITY\NETWORK SERVICE, NT AUTHORITY\LOCAL SERVICE.For more information about these limtations, check here and here.What's nextGuide for scheduling Windows containers in KubernetesManaging Workload Identity with Group Managed Service Accounts (GMSA)Configure GMSA for Windows pods and containers


# DaemonSet

DaemonSetA DaemonSet defines Pods that provide node-local facilities. These might be fundamental to the operation of your cluster, such as a networking helper tool, or be part of an add-on.A DaemonSet ensures that all (or some) Nodes run a copy of a Pod. As nodes are added to the cluster, Pods are added to them. As nodes are removed from the cluster, those Pods are garbage collected. Deleting a DaemonSet will clean up the Pods it created.Some typical uses of a DaemonSet are:running a cluster storage daemon on every noderunning a logs collection daemon on every noderunning a node monitoring daemon on every nodeIn a simple case, one DaemonSet, covering all nodes, would be used for each type of daemon. A more complex setup might use multiple DaemonSets for a single type of daemon, but with different flags and/or different memory and cpu requests for different hardware types.Writing a DaemonSet SpecCreate a DaemonSetYou can describe a DaemonSet in a YAML file. For example, the daemonset.yaml file below describes a DaemonSet that runs the fluentd-elasticsearch Docker image:controllers/daemonset.yaml apiVersion: apps/v1 kind: DaemonSet metadata: name: fluentd-elasticsearch namespace: kube-system labels: k8s-app: fluentd-logging spec: selector: matchLabels: name: fluentd-elasticsearch template: metadata: labels: name: fluentd-elasticsearch spec: tolerations: # these tolerations are to have the daemonset runnable on control plane nodes # remove them if your control plane nodes should not run pods - key: node-role.kubernetes.io/control-plane operator: Exists effect: NoSchedule - key: node-role.kubernetes.io/master operator: Exists effect: NoSchedule containers: - name: fluentd-elasticsearch image: quay.io/fluentd_elasticsearch/fluentd:v5.0.1 resources: limits: memory: 200Mi requests: cpu: 100m memory: 200Mi volumeMounts: - name: varlog mountPath: /var/log # it may be desirable to set a high priority class to ensure that a DaemonSet Pod # preempts running Pods # priorityClassName: important terminationGracePeriodSeconds: 30 volumes: - name: varlog hostPath: path: /var/log Create a DaemonSet based on the YAML file:kubectl apply -f https://k8s.io/examples/controllers/daemonset.yaml Required FieldsAs with all other Kubernetes config, a DaemonSet needs apiVersion, kind, and metadata fields. For general information about working with config files, see running stateless applications and object management using kubectl.The name of a DaemonSet object must be a valid DNS subdomain name.A DaemonSet also needs a .spec section.Pod TemplateThe .spec.template is one of the required fields in .spec.The .spec.template is a pod template. It has exactly the same schema as a Pod, except it is nested and does not have an apiVersion or kind.In addition to required fields for a Pod, a Pod template in a DaemonSet has to specify appropriate labels (see pod selector).A Pod Template in a DaemonSet must have a RestartPolicy equal to Always, or be unspecified, which defaults to Always.Pod SelectorThe .spec.selector field is a pod selector. It works the same as the .spec.selector of a Job.You must specify a pod selector that matches the labels of the .spec.template. Also, once a DaemonSet is created, its .spec.selector can not be mutated. Mutating the pod selector can lead to the unintentional orphaning of Pods, and it was found to be confusing to users.The .spec.selector is an object consisting of two fields:matchLabels - works the same as the .spec.selector of a ReplicationController.matchExpressions - allows to build more sophisticated selectors by specifying key, list of values and an operator that relates the key and values.When the two are specified the result is ANDed.The .spec.selector must match the .spec.template.metadata.labels. Config with these two not matching will be rejected by the API.Running Pods on select NodesIf you specify a .spec.template.spec.nodeSelector, then the DaemonSet controller will create Pods on nodes which match that node selector. Likewise if you specify a .spec.template.spec.affinity, then DaemonSet controller will create Pods on nodes which match that node affinity. If you do not specify either, then the DaemonSet controller will create Pods on all nodes.How Daemon Pods are scheduledA DaemonSet can be used to ensure that all eligible nodes run a copy of a Pod. The DaemonSet controller creates a Pod for each eligible node and adds the spec.affinity.nodeAffinity field of the Pod to match the target host. After the Pod is created, the default scheduler typically takes over and then binds the Pod to the target host by setting the .spec.nodeName field. If the new Pod cannot fit on the node, the default scheduler may preempt (evict) some of the existing Pods based on the priority of the new Pod.Note:If it's important that the DaemonSet pod run on each node, it's often desirable to set the .spec.template.spec.priorityClassName of the DaemonSet to a PriorityClass with a higher priority to ensure that this eviction occurs.The user can specify a different scheduler for the Pods of the DaemonSet, by setting the .spec.template.spec.schedulerName field of the DaemonSet.The original node affinity specified at the .spec.template.spec.affinity.nodeAffinity field (if specified) is taken into consideration by the DaemonSet controller when evaluating the eligible nodes, but is replaced on the created Pod with the node affinity that matches the name of the eligible node.nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchFields: - key: metadata.name operator: In values: - target-host-name Taints and tolerationsThe DaemonSet controller automatically adds a set of tolerations to DaemonSet Pods:Tolerations for DaemonSet podsToleration keyEffectDetailsnode.kubernetes.io/not-readyNoExecuteDaemonSet Pods can be scheduled onto nodes that are not healthy or ready to accept Pods. Any DaemonSet Pods running on such nodes will not be evicted.node.kubernetes.io/unreachableNoExecuteDaemonSet Pods can be scheduled onto nodes that are unreachable from the node controller. Any DaemonSet Pods running on such nodes will not be evicted.node.kubernetes.io/disk-pressureNoScheduleDaemonSet Pods can be scheduled onto nodes with disk pressure issues.node.kubernetes.io/memory-pressureNoScheduleDaemonSet Pods can be scheduled onto nodes with memory pressure issues.node.kubernetes.io/pid-pressureNoScheduleDaemonSet Pods can be scheduled onto nodes with process pressure issues.node.kubernetes.io/unschedulableNoScheduleDaemonSet Pods can be scheduled onto nodes that are unschedulable.node.kubernetes.io/network-unavailableNoScheduleOnly added for DaemonSet Pods that request host networking, i.e., Pods having spec.hostNetwork: true. Such DaemonSet Pods can be scheduled onto nodes with unavailable network.You can add your own tolerations to the Pods of a DaemonSet as well, by defining these in the Pod template of the DaemonSet.Because the DaemonSet controller sets the node.kubernetes.io/unschedulable:NoSchedule toleration automatically, Kubernetes can run DaemonSet Pods on nodes that are marked as unschedulable.If you use a DaemonSet to provide an important node-level function, such as cluster networking, it is helpful that Kubernetes places DaemonSet Pods on nodes before they are ready. For example, without that special toleration, you could end up in a deadlock situation where the node is not marked as ready because the network plugin is not running there, and at the same time the network plugin is not running on that node because the node is not yet ready.Communicating with Daemon PodsSome possible patterns for communicating with Pods in a DaemonSet are:Push: Pods in the DaemonSet are configured to send updates to another service, such as a stats database. They do not have clients.NodeIP and Known Port: Pods in the DaemonSet can use a hostPort, so that the pods are reachable via the node IPs. Clients know the list of node IPs somehow, and know the port by convention.DNS: Create a headless service with the same pod selector, and then discover DaemonSets using the endpoints resource or retrieve multiple A records from DNS.Service: Create a service with the same Pod selector, and use the service to reach a daemon on a random node. Use Service Internal Traffic Policy to limit to pods on the same node.Updating a DaemonSetIf node labels are changed, the DaemonSet will promptly add Pods to newly matching nodes and delete Pods from newly not-matching nodes.You can modify the Pods that a DaemonSet creates. However, Pods do not allow all fields to be updated. Also, the DaemonSet controller will use the original template the next time a node (even with the same name) is created.You can delete a DaemonSet. If you specify --cascade=orphan with kubectl, then the Pods will be left on the nodes. If you subsequently create a new DaemonSet with the same selector, the new DaemonSet adopts the existing Pods. If any Pods need replacing the DaemonSet replaces them according to its updateStrategy.You can perform a rolling update on a DaemonSet.Alternatives to DaemonSetInit scriptsIt is certainly possible to run daemon processes by directly starting them on a node (e.g. using init, upstartd, or systemd). This is perfectly fine. However, there are several advantages to running such processes via a DaemonSet:Ability to monitor and manage logs for daemons in the same way as applications.Same config language and tools (e.g. Pod templates, kubectl) for daemons and applications.Running daemons in containers with resource limits increases isolation between daemons from app containers. However, this can also be accomplished by running the daemons in a container but not in a Pod.Bare PodsIt is possible to create Pods directly which specify a particular node to run on. However, a DaemonSet replaces Pods that are deleted or terminated for any reason, such as in the case of node failure or disruptive node maintenance, such as a kernel upgrade. For this reason, you should use a DaemonSet rather than creating individual Pods.Static PodsIt is possible to create Pods by writing a file to a certain directory watched by Kubelet. These are called static pods. Unlike DaemonSet, static Pods cannot be managed with kubectl or other Kubernetes API clients. Static Pods do not depend on the apiserver, making them useful in cluster bootstrapping cases. Also, static Pods may be deprecated in the future.DeploymentsDaemonSets are similar to Deployments in that they both create Pods, and those Pods have processes which are not expected to terminate (e.g. web servers, storage servers).Use a Deployment for stateless services, like frontends, where scaling up and down the number of replicas and rolling out updates are more important than controlling exactly which host the Pod runs on. Use a DaemonSet when it is important that a copy of a Pod always run on all or certain hosts, if the DaemonSet provides node-level functionality that allows other Pods to run correctly on that particular node.For example, network plugins often include a component that runs as a DaemonSet. The DaemonSet component makes sure that the node where it's running has working cluster networking.What's nextLearn about Pods.Learn about static Pods, which are useful for running Kubernetes control plane components.Find out how to use DaemonSetsPerform a rolling update on a DaemonSetPerform a rollback on a DaemonSet (for example, if a roll out didn't work how you expected).Understand how Kubernetes assigns Pods to Nodes.Learn about device plugins and add ons, which often run as DaemonSets.DaemonSet is a top-level resource in the Kubernetes REST API. Read the DaemonSet object definition to understand the API for daemon sets.


# Configure a Pod to Use a PersistentVolume for Storage

Configure a Pod to Use a PersistentVolume for StorageThis page shows you how to configure a Pod to use a PersistentVolumeClaim for storage. Here is a summary of the process:You, as cluster administrator, create a PersistentVolume backed by physical storage. You do not associate the volume with any Pod.You, now taking the role of a developer / cluster user, create a PersistentVolumeClaim that is automatically bound to a suitable PersistentVolume.You create a Pod that uses the above PersistentVolumeClaim for storage.Before you beginYou need to have a Kubernetes cluster that has only one Node, and the kubectl command-line tool must be configured to communicate with your cluster. If you do not already have a single-node cluster, you can create one by using Minikube.Familiarize yourself with the material in Persistent Volumes.Create an index.html file on your NodeOpen a shell to the single Node in your cluster. How you open a shell depends on how you set up your cluster. For example, if you are using Minikube, you can open a shell to your Node by entering minikube ssh.In your shell on that Node, create a /mnt/data directory:# This assumes that your Node uses "sudo" to run commands # as the superuser sudo mkdir /mnt/data In the /mnt/data directory, create an index.html file:# This again assumes that your Node uses "sudo" to run commands # as the superuser sudo sh -c "echo 'Hello from Kubernetes storage' > /mnt/data/index.html" Note:If your Node uses a tool for superuser access other than sudo, you can usually make this work if you replace sudo with the name of the other tool.Test that the index.html file exists:cat /mnt/data/index.html The output should be:Hello from Kubernetes storage You can now close the shell to your Node.Create a PersistentVolumeIn this exercise, you create a hostPath PersistentVolume. Kubernetes supports hostPath for development and testing on a single-node cluster. A hostPath PersistentVolume uses a file or directory on the Node to emulate network-attached storage.In a production cluster, you would not use hostPath. Instead a cluster administrator would provision a network resource like a Google Compute Engine persistent disk, an NFS share, or an Amazon Elastic Block Store volume. Cluster administrators can also use StorageClasses to set up dynamic provisioning.Here is the configuration file for the hostPath PersistentVolume:pods/storage/pv-volume.yaml apiVersion: v1 kind: PersistentVolume metadata: name: task-pv-volume labels: type: local spec: storageClassName: manual capacity: storage: 10Gi accessModes: - ReadWriteOnce hostPath: path: "/mnt/data" The configuration file specifies that the volume is at /mnt/data on the cluster's Node. The configuration also specifies a size of 10 gibibytes and an access mode of ReadWriteOnce, which means the volume can be mounted as read-write by a single Node. It defines the StorageClass name manual for the PersistentVolume, which will be used to bind PersistentVolumeClaim requests to this PersistentVolume.Note:This example uses the ReadWriteOnce access mode, for simplicity. For production use, the Kubernetes project recommends using the ReadWriteOncePod access mode instead.Create the PersistentVolume:kubectl apply -f https://k8s.io/examples/pods/storage/pv-volume.yaml View information about the PersistentVolume:kubectl get pv task-pv-volume The output shows that the PersistentVolume has a STATUS of Available. This means it has not yet been bound to a PersistentVolumeClaim.NAME CAPACITY ACCESSMODES RECLAIMPOLICY STATUS CLAIM STORAGECLASS REASON AGE task-pv-volume 10Gi RWO Retain Available manual 4s Create a PersistentVolumeClaimThe