Virtual IPs and Service ProxiesEvery node in a Kubernetes cluster runs a kube-proxy (unless you have deployed your own alternative component in place of kube-proxy).The kube-proxy component is responsible for implementing a virtual IP mechanism for Services of type other than ExternalName. Each instance of kube-proxy watches the Kubernetes control plane for the addition and removal of Service and EndpointSlice objects. For each Service, kube-proxy calls appropriate APIs (depending on the kube-proxy mode) to configure the node to capture traffic to the Service's clusterIP and port, and redirect that traffic to one of the Service's endpoints (usually a Pod, but possibly an arbitrary user-provided IP address). A control loop ensures that the rules on each node are reliably synchronized with the Service and EndpointSlice state as indicated by the API server.Virtual IP mechanism for Services, using iptables modeA question that pops up every now and then is why Kubernetes relies on proxying to forward inbound traffic to backends. What about other approaches? For example, would it be possible to configure DNS records that have multiple A values (or AAAA for IPv6), and rely on round-robin name resolution?There are a few reasons for using proxying for Services:There is a long history of DNS implementations not respecting record TTLs, and caching the results of name lookups after they should have expired.Some apps do DNS lookups only once and cache the results indefinitely.Even if apps and libraries did proper re-resolution, the low or zero TTLs on the DNS records could impose a high load on DNS that then becomes difficult to manage.Later in this page you can read about how various kube-proxy implementations work. Overall, you should note that, when running kube-proxy, kernel level rules may be modified (for example, iptables rules might get created), which won't get cleaned up, in some cases until you reboot. Thus, running kube-proxy is something that should only be done by an administrator who understands the consequences of having a low level, privileged network proxying service on a computer. Although the kube-proxy executable supports a cleanup function, this function is not an official feature and thus is only available to use as-is.Some of the details in this reference refer to an example: the backend Pods for a stateless image-processing workloads, running with three replicas. Those replicas are fungible—frontends do not care which backend they use. While the actual Pods that compose the backend set may change, the frontend clients should not need to be aware of that, nor should they need to keep track of the set of backends themselves.Proxy modesThe kube-proxy starts up in different modes, which are determined by its configuration.On Linux nodes, the available modes for kube-proxy are:iptablesA mode where the kube-proxy configures packet forwarding rules using iptables.ipvsa mode where the kube-proxy configures packet forwarding rules using ipvs.nftablesa mode where the kube-proxy configures packet forwarding rules using nftables.There is only one mode available for kube-proxy on Windows:kernelspacea mode where the kube-proxy configures packet forwarding rules in the Windows kerneliptables proxy modeThis proxy mode is only available on Linux nodes.In this mode, kube-proxy configures packet forwarding rules using the iptables API of the kernel netfilter subsystem. For each endpoint, it installs iptables rules which, by default, select a backend Pod at random.ExampleAs an example, consider the image processing application described earlier in the page. When the backend Service is created, the Kubernetes control plane assigns a virtual IP address, for example 10.0.0.1. For this example, assume that the Service port is 1234. All of the kube-proxy instances in the cluster observe the creation of the new Service.When kube-proxy on a node sees a new Service, it installs a series of iptables rules which redirect from the virtual IP address to more iptables rules, defined per Service. The per-Service rules link to further rules for each backend endpoint, and the per- endpoint rules redirect traffic (using destination NAT) to the backends.When a client connects to the Service's virtual IP address the iptables rule kicks in. A backend is chosen (either based on session affinity or randomly) and packets are redirected to the backend without rewriting the client IP address.This same basic flow executes when traffic comes in through a type: NodePort Service, or through a load-balancer, though in those cases the client IP address does get altered.Optimizing iptables mode performanceIn iptables mode, kube-proxy creates a few iptables rules for every Service, and a few iptables rules for each endpoint IP address. In clusters with tens of thousands of Pods and Services, this means tens of thousands of iptables rules, and kube-proxy may take a long time to update the rules in the kernel when Services (or their EndpointSlices) change. You can adjust the syncing behavior of kube-proxy via options in the iptables section of the kube-proxy configuration file (which you specify via kube-proxy --config <path>):... iptables: minSyncPeriod: 1s syncPeriod: 30s ... minSyncPeriodThe minSyncPeriod parameter sets the minimum duration between attempts to resynchronize iptables rules with the kernel. If it is 0s, then kube-proxy will always immediately synchronize the rules every time any Service or EndpointSlice changes. This works fine in very small clusters, but it results in a lot of redundant work when lots of things change in a small time period. For example, if you have a Service backed by a Deployment with 100 pods, and you delete the Deployment, then with minSyncPeriod: 0s, kube-proxy would end up removing the Service's endpoints from the iptables rules one by one, resulting in a total of 100 updates. With a larger minSyncPeriod, multiple Pod deletion events would get aggregated together, so kube-proxy might instead end up making, say, 5 updates, each removing 20 endpoints, which will be much more efficient in terms of CPU, and result in the full set of changes being synchronized faster.The larger the value of minSyncPeriod, the more work that can be aggregated, but the downside is that each individual change may end up waiting up to the full minSyncPeriod before being processed, meaning that the iptables rules spend more time being out-of-sync with the current API server state.The default value of 1s should work well in most clusters, but in very large clusters it may be necessary to set it to a larger value. Especially, if kube-proxy's sync_proxy_rules_duration_seconds metric indicates an average time much larger than 1 second, then bumping up minSyncPeriod may make updates more efficient.Updating legacy minSyncPeriod configurationOlder versions of kube-proxy updated all the rules for all Services on every sync; this led to performance issues (update lag) in large clusters, and the recommended solution was to set a larger minSyncPeriod. Since Kubernetes v1.28, the iptables mode of kube-proxy uses a more minimal approach, only making updates where Services or EndpointSlices have actually changed.If you were previously overriding minSyncPeriod, you should try removing that override and letting kube-proxy use the default value (1s) or at least a smaller value than you were using before upgrading.If you are not running kube-proxy from Kubernetes 1.34, check the behavior and associated advice for the version that you are actually running.syncPeriodThe syncPeriod parameter controls a handful of synchronization operations that are not directly related to changes in individual Services and EndpointSlices. In particular, it controls how quickly kube-proxy notices if an external component has interfered with kube-proxy's iptables rules. In large clusters, kube-proxy also only performs certain cleanup operations once every syncPeriod to avoid unnecessary work.For the most part, increasing syncPeriod is not expected to have much impact on performance, but in the past, it was sometimes useful to set it to a very large value (eg, 1h). This is no longer recommended, and is likely to hurt functionality more than it improves performance.IPVS proxy modeThis proxy mode is only available on Linux nodes.In ipvs mode, kube-proxy uses the kernel IPVS and iptables APIs to create rules to redirect traffic from Service IPs to endpoint IPs.The IPVS proxy mode is based on netfilter hook function that is similar to iptables mode, but uses a hash table as the underlying data structure and works in the kernel space.Note:The ipvs proxy mode was an experiment in providing a Linux kube-proxy backend with better rule-synchronizing performance and higher network-traffic throughput than the iptables mode. While it succeeded in those goals, the kernel IPVS API turned out to be a bad match for the Kubernetes Services API, and the ipvs backend was never able to implement all of the edge cases of Kubernetes Service functionality correctly. At some point in the future, it is expected to be formally deprecated as a feature.The nftables proxy mode (described below) is essentially a replacement for both the iptables and ipvs modes, with better performance than either of them, and is recommended as a replacement for ipvs. If you are deploying onto Linux systems that are too old to run the nftables proxy mode, you should also consider trying the iptables mode rather than ipvs, since the performance of iptables mode has improved greatly since the ipvs mode was first introduced.IPVS provides more options for balancing traffic to backend Pods; these are:rr (Round Robin): Traffic is equally distributed amongst the backing servers.wrr (Weighted Round Robin): Traffic is routed to the backing servers based on the weights of the servers. Servers with higher weights receive new connections and get more requests than servers with lower weights.lc (Least Connection): More traffic is assigned to servers with fewer active connections.wlc (Weighted Least Connection): More traffic is routed to servers with fewer connections relative to their weights, that is, connections divided by weight.lblc (Locality based Least Connection): Traffic for the same IP address is sent to the same backing server if the server is not overloaded and available; otherwise the traffic is sent to servers with fewer connections, and keep it for future assignment.lblcr (Locality Based Least Connection with Replication): Traffic for the same IP address is sent to the server with least connections. If all the backing servers are overloaded, it picks up one with fewer connections and adds it to the target set. If the target set has not changed for the specified time, the server with the highest load is removed from the set, in order to avoid a high degree of replication.sh (Source Hashing): Traffic is sent to a backing server by looking up a statically assigned hash table based on the source IP addresses.dh (Destination Hashing): Traffic is sent to a backing server by looking up a statically assigned hash table based on their destination addresses.sed (Shortest Expected Delay): Traffic forwarded to a backing server with the shortest expected delay. The expected delay is (C + 1) / U if sent to a server, where C is the number of connections on the server and U is the fixed service rate (weight) of the server.nq (Never Queue): Traffic is sent to an idle server if there is one, instead of waiting for a fast one; if all servers are busy, the algorithm falls back to the sed behavior.mh (Maglev Hashing): Assigns incoming jobs based on Google's Maglev hashing algorithm, This scheduler has two flags: mh-fallback, which enables fallback to a different server if the selected server is unavailable, and mh-port, which adds the source port number to the hash computation. When using mh, kube-proxy always sets the mh-port flag and does not enable the mh-fallback flag. In proxy-mode=ipvs mh will work as source-hashing (sh), but with ports.These scheduling algorithms are configured through the ipvs.scheduler field in the kube-proxy configuration.Note:To run kube-proxy in IPVS mode, you must make IPVS available on the node before starting kube-proxy.When kube-proxy starts in IPVS proxy mode, it verifies whether IPVS kernel modules are available. If the IPVS kernel modules are not detected, then kube-proxy exits with an error.Virtual IP address mechanism for Services, using IPVS modenftables proxy modeFEATURE STATE: Kubernetes v1.33 [stable] (enabled by default: true)This proxy mode is only available on Linux nodes, and requires kernel 5.13 or later.In this mode, kube-proxy configures packet forwarding rules using the nftables API of the kernel netfilter subsystem. For each endpoint, it installs nftables rules which, by default, select a backend Pod at random.The nftables API is the successor to the iptables API and is designed to provide better performance and scalability than iptables. The nftables proxy mode is able to process changes to service endpoints faster and more efficiently than the iptables mode, and is also able to more efficiently process packets in the kernel (though this only becomes noticeable in clusters with tens of thousands of services).As of Kubernetes 1.34, the nftables mode is still relatively new, and may not be compatible with all network plugins; consult the documentation for your network plugin.Migrating from iptables mode to nftablesUsers who want to switch from the default iptables mode to the nftables mode should be aware that some features work slightly differently the nftables mode:NodePort interfaces: In iptables mode, by default, NodePort services are reachable on all local IP addresses. This is usually not what users want, so the nftables mode defaults to --nodeport-addresses primary, meaning Services using type: NodePort are only reachable on the node's primary IPv4 and/or IPv6 addresses. You can override this by specifying an explicit value for that option: e.g., --nodeport-addresses 0.0.0.0/0 to listen on all (local) IPv4 IPs.type: NodePort Services on 127.0.0.1: In iptables mode, if the --nodeport-addresses range includes 127.0.0.1 (and the option --iptables-localhost-nodeports false option is not passed), then Services of type: NodePort are reachable even on "localhost" (127.0.0.1). In nftables mode (and ipvs mode), this will not work. If you are not sure if you are depending on this functionality, you can check kube-proxy's iptables_localhost_nodeports_accepted_packets_total metric; if it is non-0, that means that some client has connected to a type: NodePort Service via localhost/loopback.NodePort interaction with firewalls: The iptables mode of kube-proxy tries to be compatible with overly-aggressive firewalls; for each type: NodePort service, it will add rules to accept inbound traffic on that port, in case that traffic would otherwise be blocked by a firewall. This approach will not work with firewalls based on nftables, so kube-proxy's nftables mode does not do anything here; if you have a local firewall, you must ensure that it is properly configured to allow Kubernetes traffic through (e.g., by allowing inbound traffic on the entire NodePort range).Conntrack bug workarounds: Linux kernels prior to 6.1 have a bug that can result in long-lived TCP connections to service IPs being closed with the error "Connection reset by peer". The iptables mode of kube-proxy installs a workaround for this bug, but this workaround was later found to cause other problems in some clusters. The nftables mode does not install any workaround by default, but you can check kube-proxy's iptables_ct_state_invalid_dropped_packets_total metric to see if your cluster is depending on the workaround, and if so, you can run kube-proxy with the option --conntrack-tcp-be-liberal to work around the problem in nftables mode.kernelspace proxy modeThis proxy mode is only available on Windows nodes.The kube-proxy configures packet filtering rules in the Windows Virtual Filtering Platform (VFP), an extension to Windows vSwitch. These rules process encapsulated packets within the node-level virtual networks, and rewrite packets so that the destination IP address (and layer 2 information) is correct for getting the packet routed to the correct destination. The Windows VFP is analogous to tools such as Linux nftables or iptables. The Windows VFP extends the Hyper-V Switch, which was initially implemented to support virtual machine networking.When a Pod on a node sends traffic to a virtual IP address, and the kube-proxy selects a Pod on a different node as the load balancing target, the kernelspace proxy mode rewrites that packet to be destined to the target backend Pod. The Windows Host Networking Service (HNS) ensures that packet rewriting rules are configured so that the return traffic appears to come from the virtual IP address and not the specific backend Pod.Direct server return for kernelspace modeFEATURE STATE: Kubernetes v1.34 [stable] (enabled by default: true)As an alternative to the basic operation, a node that hosts the backend Pod for a Service can apply the packet rewriting directly, rather than placing this burden on the node where the client Pod is running. This is called direct server return.To use this, you must run kube-proxy with the --enable-dsr command line argument and enable the WinDSR feature gate.Direct server return also optimizes the case for Pod return traffic even when both Pods are running on the same node.Session affinityIn these proxy models, the traffic bound for the Service's IP:Port is proxied to an appropriate backend without the clients knowing anything about Kubernetes or Services or Pods.If you want to make sure that connections from a particular client are passed to the same Pod each time, you can select the session affinity based on the client's IP addresses by setting .spec.sessionAffinity to ClientIP for a Service (the default is None).Session stickiness timeoutYou can also set the maximum session sticky time by setting .spec.sessionAffinityConfig.clientIP.timeoutSeconds appropriately for a Service. (the default value is 10800, which works out to be 3 hours).Note:On Windows, setting the maximum session sticky time for Services is not supported.IP address assignment to ServicesUnlike Pod IP addresses, which actually route to a fixed destination, Service IPs are not actually answered by a single host. Instead, kube-proxy uses packet processing logic (such as Linux iptables) to define virtual IP addresses which are transparently redirected as needed.When clients connect to the VIP, their traffic is automatically transported to an appropriate endpoint. The environment variables and DNS for Services are actually populated in terms of the Service's virtual IP address (and port).Avoiding collisionsOne of the primary philosophies of Kubernetes is that you should not be exposed to situations that could cause your actions to fail through no fault of your own. For the design of the Service resource, this means not making you choose your own IP address if that choice might collide with someone else's choice. That is an isolation failure.In order to allow you to choose an IP address for your Services, we must ensure that no two Services can collide. Kubernetes does that by allocating each Service its own IP address from within the service-cluster-ip-range CIDR range that is configured for the API Server.IP address allocation trackingTo ensure each Service receives a unique IP address, an internal allocator atomically updates a global allocation map in etcd prior to creating each Service. The map object must exist in the registry for Services to get IP address assignments, otherwise creations will fail with a message indicating an IP address could not be allocated.In the control plane, a background controller is responsible for creating that map (needed to support migrating from older versions of Kubernetes that used in-memory locking). Kubernetes also uses controllers to check for invalid assignments (for example: due to administrator intervention) and for cleaning up allocated IP addresses that are no longer used by any Services.IP address allocation tracking using the Kubernetes APIFEATURE STATE: Kubernetes v1.33 [stable] (enabled by default: true)The control plane replaces the existing etcd allocator with a revised implementation that uses IPAddress and ServiceCIDR objects instead of an internal global allocation map. Each cluster IP address associated to a Service then references an IPAddress object.Enabling the feature gate also replaces a background controller with an alternative that handles the IPAddress objects and supports migration from the old allocator model. Kubernetes 1.34 does not support migrating from IPAddress objects to the internal allocation map.One of the main benefits of the revised allocator is that it removes the size limitations for the IP address range that can be used for the cluster IP address of Services. With MultiCIDRServiceAllocator enabled, there are no limitations for IPv4, and for IPv6 you can use IP address netmasks that are a /64 or smaller (as opposed to /108 with the legacy implementation).Making IP address allocations available via the API means that you as a cluster administrator can allow users to inspect the IP addresses assigned to their Services. Kubernetes extensions, such as the Gateway API, can use the IPAddress API to extend Kubernetes' inherent networking capabilities.Here is a brief example of a user querying for IP addresses:kubectl get services NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 2001:db8:1:2::1 <none> 443/TCP 3d1h kubectl get ipaddresses NAME PARENTREF 2001:db8:1:2::1 services/default/kubernetes 2001:db8:1:2::a services/kube-system/kube-dns Kubernetes also allow users to dynamically define the available IP ranges for Services using ServiceCIDR objects. During bootstrap, a default ServiceCIDR object named kubernetes is created from the value of the --service-cluster-ip-range command line argument to kube-apiserver:kubectl get servicecidrs NAME CIDRS AGE kubernetes 10.96.0.0/28 17m Users can create or delete new ServiceCIDR objects to manage the available IP ranges for Services:cat <<'EOF' | kubectl apply -f - apiVersion: networking.k8s.io/v1 kind: ServiceCIDR metadata: name: newservicecidr spec: cidrs: - 10.96.0.0/24 EOF servicecidr.networking.k8s.io/newcidr1 created kubectl get servicecidrs NAME CIDRS AGE kubernetes 10.96.0.0/28 17m newservicecidr 10.96.0.0/24 7m Distributions or administrators of Kubernetes clusters may want to control that new Service CIDRs added to the cluster does not overlap with other networks on the cluster, that only belong to a specific range of IPs or just simple retain the existing behavior of only having one ServiceCIDR per cluster. An example of a Validation Admission Policy to achieve this is:--- apiVersion: admissionregistration.k8s.io/v1 kind: ValidatingAdmissionPolicy metadata: name: "servicecidrs-default" spec: failurePolicy: Fail matchConstraints: resourceRules: - apiGroups: ["networking.k8s.io"] apiVersions: ["v1","v1beta1"] operations: ["CREATE", "UPDATE"] resources: ["servicecidrs"] matchConditions: - name: 'exclude-default-servicecidr' expression: "object.metadata.name != 'kubernetes'" variables: - name: allowed expression: "['10.96.0.0/16','2001:db8::/64']" validations: - expression: "object.spec.cidrs.all(i , variables.allowed.exists(j , cidr(j).containsCIDR(i)))" --- apiVersion: admissionregistration.k8s.io/v1 kind: ValidatingAdmissionPolicyBinding metadata: name: "servicecidrs-binding" spec: policyName: "servicecidrs-default" validationActions: [Deny,Audit] --- IP address ranges for Service virtual IP addressesFEATURE STATE: Kubernetes v1.26 [stable]Kubernetes divides the ClusterIP range into two bands, based on the size of the configured service-cluster-ip-range by using the following formula min(max(16, cidrSize / 16), 256). That formula means the result is never less than 16 or more than 256, with a graduated step function between them.Kubernetes prefers to allocate dynamic IP addresses to Services by choosing from the upper band, which means that if you want to assign a specific IP address to a type: ClusterIP Service, you should manually assign an IP address from the lower band. That approach reduces the risk of a conflict over allocation.Traffic policiesYou can set the .spec.internalTrafficPolicy and .spec.externalTrafficPolicy fields to control how Kubernetes routes traffic to healthy (“ready”) backends.Internal traffic policyFEATURE STATE: Kubernetes v1.26 [stable]You can set the .spec.internalTrafficPolicy field to control how traffic from internal sources is routed. Valid values are Cluster and Local. Set the field to Cluster to route internal traffic to all ready endpoints and Local to only route to ready node-local endpoints. If the traffic policy is Local and there are no node-local endpoints, traffic is dropped by kube-proxy.External traffic policyYou can set the .spec.externalTrafficPolicy field to control how traffic from external sources is routed. Valid values are Cluster and Local. Set the field to Cluster to route external traffic to all ready endpoints and Local to only route to ready node-local endpoints. If the traffic policy is Local and there are no node-local endpoints, the kube-proxy does not forward any traffic for the relevant Service.If Cluster is specified, all nodes are eligible load balancing targets as long as the node is not being deleted and kube-proxy is healthy. In this mode: load balancer health checks are configured to target the service proxy's readiness port and path. In the case of kube-proxy this evaluates to: ${NODE_IP}:10256/healthz. kube-proxy will return either an HTTP code 200 or 503. kube-proxy's load balancer health check endpoint returns 200 if:kube-proxy is healthy, meaning:it's able to progress programming the network and isn't timing out while doing so (the timeout is defined to be: 2 × iptables.syncPeriod); andthe node is not being deleted (there is no deletion timestamp set for the Node).kube-proxy returns 503 and marks the node as not eligible when it's being deleted because it supports connection draining for terminating nodes. A couple of important things occur from the point of view of a Kubernetes-managed load balancer when a node is being / is deleted.While deleting:kube-proxy will start failing its readiness probe and essentially mark the node as not eligible for load balancer traffic. The load balancer health check failing causes load balancers which support connection draining to allow existing connections to terminate, and block new connections from establishing.When deleted:The service controller in the Kubernetes cloud controller manager removes the node from the referenced set of eligible targets. Removing any instance from the load balancer's set of backend targets immediately terminates all connections. This is also the reason kube-proxy first fails the health check while the node is deleting.It's important to note for Kubernetes vendors that if any vendor configures the kube-proxy readiness probe as a liveness probe: that kube-proxy will start restarting continuously when a node is deleting until it has been fully deleted. kube-proxy exposes a /livez path which, as opposed to the /healthz one, does not consider the Node's deleting state and only its progress programming the network. /livez is therefore the recommended path for anyone looking to define a livenessProbe for kube-proxy.Users deploying kube-proxy can inspect both the readiness / liveness state by evaluating the metrics: proxy_livez_total / proxy_healthz_total. Both metrics publish two series, one with the 200 label and one with the 503 one.For Local Services: kube-proxy will return 200 ifkube-proxy is healthy/ready, andhas a local endpoint on the node in question.Node deletion does not have an impact on kube-proxy's return code for what concerns load balancer health checks. The reason for this is: deleting nodes could end up causing an ingress outage should all endpoints simultaneously be running on said nodes.The Kubernetes project recommends that cloud provider integration code configures load balancer health checks that target the service proxy's healthz port. If you are using or implementing your own virtual IP implementation, that people can use instead of kube-proxy, you should set up a similar health checking port with logic that matches the kube-proxy implementation.Traffic to terminating endpointsFEATURE STATE: Kubernetes v1.28 [stable]If the ProxyTerminatingEndpoints feature gate is enabled in kube-proxy and the traffic policy is Local, that node's kube-proxy uses a more complicated algorithm to select endpoints for a Service. With the feature enabled, kube-proxy checks if the node has local endpoints and whether or not all the local endpoints are marked as terminating. If there are local endpoints and all of them are terminating, then kube-proxy will forward traffic to those terminating endpoints. Otherwise, kube-proxy will always prefer forwarding traffic to endpoints that are not terminating.This forwarding behavior for terminating endpoints exists to allow NodePort and LoadBalancer Services to gracefully drain connections when using externalTrafficPolicy: Local.As a deployment goes through a rolling update, nodes backing a load balancer may transition from N to 0 replicas of that deployment. In some cases, external load balancers can send traffic to a node with 0 replicas in between health check probes. Routing traffic to terminating endpoints ensures that Nodes that are scaling down Pods can gracefully receive and drain traffic to those terminating Pods. By the time the Pod completes termination, the external load balancer should have seen the node's health check failing and fully removed the node from the backend pool.Traffic DistributionFEATURE STATE: Kubernetes v1.33 [stable] (enabled by default: true)The spec.trafficDistribution field within a Kubernetes Service allows you to express preferences for how traffic should be routed to Service endpoints.PreferCloseThis prioritizes sending traffic to endpoints in the same zone as the client. The EndpointSlice controller updates EndpointSlices with hints to communicate this preference, which kube-proxy then uses for routing decisions. If a client's zone does not have any available endpoints, traffic will be routed cluster-wide for that client.FEATURE STATE: Kubernetes v1.34 [beta] (enabled by default: true)In Kubernetes 1.34, two additional values are available (unless the PreferSameTrafficDistribution feature gate is disabled):PreferSameZoneThis means the same thing as PreferClose, but is more explicit. (Originally, the intention was that PreferClose might later include functionality other than just "prefer same zone", but this is no longer planned. In the future, PreferSameZone will be the recommended value to use for this functionality, and PreferClose will be considered a deprecated alias for it.)PreferSameNodeThis prioritizes sending traffic to endpoints on the same node as the client. As with PreferClose/PreferSameZone, the EndpointSlice controller updates EndpointSlices with hints indicating that a slice should be used for a particular node. If a client's node does not have any available endpoints, then the service proxy will fall back to "same zone" behavior, or cluster-wide if there are no same-zone endpoints either.In the absence of any value for trafficDistribution, the default strategy is to distribute traffic evenly to all endpoints in the cluster.Comparison with service.kubernetes.io/topology-mode: AutoThe trafficDistribution field with PreferClose/PreferSameZone, and the older "Topology-Aware Routing" feature using the service.kubernetes.io/topology-mode: Auto annotation both aim to prioritize same-zone traffic. However, there is a key difference in their approaches:service.kubernetes.io/topology-mode: Auto attempts to distribute traffic proportionally across zones based on allocatable CPU resources. This heuristic includes safeguards (such as the fallback behavior for small numbers of endpoints), sacrificing some predictability in favor of potentially better load balancing.trafficDistribution: PreferClose aims to be simpler and more predictable: "If there are endpoints in the zone, they will receive all traffic for that zone, if there are no endpoints in a zone, the traffic will be distributed to other zones". This approach offers more predictability, but it means that you are responsible for avoiding endpoint overload.If the service.kubernetes.io/topology-mode annotation is set to Auto, it will take precedence over trafficDistribution. The annotation may be deprecated in the future in favor of the trafficDistribution field.Interaction with Traffic PoliciesWhen compared to the trafficDistribution field, the traffic policy fields (externalTrafficPolicy and internalTrafficPolicy) are meant to offer a stricter traffic locality requirements. Here's how trafficDistribution interacts with them:Precedence of Traffic Policies: For a given Service, if a traffic policy (externalTrafficPolicy or internalTrafficPolicy) is set to Local, it takes precedence over trafficDistribution for the corresponding traffic type (external or internal, respectively).trafficDistribution Influence: For a given Service, if a traffic policy (externalTrafficPolicy or internalTrafficPolicy) is set to Cluster (the default), or if the fields are not set, then trafficDistribution guides the routing behavior for the corresponding traffic type (external or internal, respectively). This means that an attempt will be made to route traffic to an endpoint that is in the same zone as the client.Considerations for using traffic distribution controlA Service using trafficDistribution will attempt to route traffic to (healthy) endpoints within the appropriate topology, even if this means that some endpoints receive much more traffic than other endpoints. If you do not have a sufficient number of endpoints within the same topology ("same zone", "same node", etc.) as the clients, then endpoints may become overloaded. This is especially likely if incoming traffic is not proportionally distributed across the topology. To mitigate this, consider the following strategies:Pod Topology Spread Constraints: Use Pod Topology Spread Constraints to distribute your pods evenly across zones or nodes.Zone-specific Deployments: If you are using "same zone" traffic distribution, but expect to see different traffic patterns in different zones, you can create a separate Deployment for each zone. This approach allows the separate workloads to scale independently. There are also workload management addons available from the ecosystem, outside the Kubernetes project itself, that can help here.What's nextTo learn more about Services, read Connecting Applications with Services.You can also:Read about Services as a conceptRead about Ingresses as a conceptRead the API reference for the Service API

Virtual IPs and Service Proxies


Common Expression Language in KubernetesThe Common Expression Language (CEL) is used in the Kubernetes API to declare validation rules, policy rules, and other constraints or conditions.CEL expressions are evaluated directly in the API server, making CEL a convenient alternative to out-of-process mechanisms, such as webhooks, for many extensibility use cases. Your CEL expressions continue to execute so long as the control plane's API server component remains available.Language overviewThe CEL language has a straightforward syntax that is similar to the expressions in C, C++, Java, JavaScript and Go.CEL was designed to be embedded into applications. Each CEL "program" is a single expression that evaluates to a single value. CEL expressions are typically short "one-liners" that inline well into the string fields of Kubernetes API resources.Inputs to a CEL program are "variables". Each Kubernetes API field that contains CEL declares in the API documentation which variables are available to use for that field. For example, in the x-kubernetes-validations[i].rules field of CustomResourceDefinitions, the self and oldSelf variables are available and refer to the previous and current state of the custom resource data to be validated by the CEL expression. Other Kubernetes API fields may declare different variables. See the API documentation of the API fields to learn which variables are available for that field.Example CEL expressions:Examples of CEL expressions and the purpose of eachRulePurposeself.minReplicas <= self.replicas && self.replicas <= self.maxReplicasValidate that the three fields defining replicas are ordered appropriately'Available' in self.stateCountsValidate that an entry with the 'Available' key exists in a map(self.list1.size() == 0) != (self.list2.size() == 0)Validate that one of two lists is non-empty, but not bothself.envars.filter(e, e.name = 'MY_ENV').all(e, e.value.matches('^[a-zA-Z]*$'))Validate the 'value' field of a listMap entry where key field 'name' is 'MY_ENV'has(self.expired) && self.created + self.ttl < self.expiredValidate that 'expired' date is after a 'create' date plus a 'ttl' durationself.health.startsWith('ok')Validate a 'health' string field has the prefix 'ok'self.widgets.exists(w, w.key == 'x' && w.foo < 10)Validate that the 'foo' property of a listMap item with a key 'x' is less than 10type(self) == string ? self == '99%' : self == 42Validate an int-or-string field for both the int and string casesself.metadata.name == 'singleton'Validate that an object's name matches a specific value (making it a singleton)self.set1.all(e, !(e in self.set2))Validate that two listSets are disjointself.names.size() == self.details.size() && self.names.all(n, n in self.details)Validate the 'details' map is keyed by the items in the 'names' listSetself.details.all(key, key.matches('^[a-zA-Z]*$'))Validate the keys of the 'details' mapself.details.all(key, self.details[key].matches('^[a-zA-Z]*$'))Validate the values of the 'details' mapCEL options, language features, and librariesCEL is configured with the following options, libraries and language features, introduced at the specified Kubernetes versions:CEL option, library or language featureIncludedAvailabilityStandard macroshas, all, exists, exists_one, map, filterAll Kubernetes versionsStandard functionsSee official list of standard definitionsAll Kubernetes versionsHomogeneous Aggregate Literals-All Kubernetes versionsDefault UTC Time Zone-All Kubernetes versionsEagerly Validate Declarations-All Kubernetes versionsExtended strings library, Version 1charAt, indexOf, lastIndexOf, lowerAscii, upperAscii, replace, split, join, substring, trimKubernetes versions between 1.25 and 1.30Extended strings library, Version 2charAt, indexOf, lastIndexOf, lowerAscii, upperAscii, replace, split, join, substring, trimKubernetes versions 1.30+Kubernetes list librarySee Kubernetes list libraryAll Kubernetes versionsKubernetes regex librarySee Kubernetes regex libraryAll Kubernetes versionsKubernetes URL librarySee Kubernetes URL libraryAll Kubernetes versionsKubernetes IP address librarySee Kubernetes IP address libraryKubernetes versions 1.31+Kubernetes CIDR librarySee Kubernetes CIDR libraryKubernetes versions 1.31+Kubernetes authorizer librarySee Kubernetes authorizer libraryAll Kubernetes versionsKubernetes quantity librarySee Kubernetes quantity libraryKubernetes versions 1.29+Kubernetes semver librarySee Kubernetes semver libraryKubernetes versions 1.34+Kubernetes format librarySee Kubernetes format libraryKubernetes versions 1.32+CEL optional typesSee CEL optional typesKubernetes versions 1.29+CEL CrossTypeNumericComparisonsSee CEL CrossTypeNumericComparisonsKubernetes versions 1.29+CEL TwoVarComprehensionsSee CEL TwoVarComprehensionsKubernetes versions 1.33+CEL functions, features and language settings support Kubernetes control plane rollbacks. For example, CEL Optional Values was introduced at Kubernetes 1.29 and so only API servers at that version or newer will accept write requests to CEL expressions that use CEL Optional Values. However, when a cluster is rolled back to Kubernetes 1.28 CEL expressions using "CEL Optional Values" that are already stored in API resources will continue to evaluate correctly.Kubernetes CEL librariesIn additional to the CEL community libraries, Kubernetes includes CEL libraries that are available everywhere CEL is used in Kubernetes.Kubernetes list libraryThe list library includes indexOf and lastIndexOf, which work similar to the strings functions of the same names. These functions either the first or last positional index of the provided element in the list.The list library also includes min, max and sum. Sum is supported on all number types as well as the duration type. Min and max are supported on all comparable types.isSorted is also provided as a convenience function and is supported on all comparable types.Examples:Examples of CEL expressions using list library functionsCEL ExpressionPurposenames.isSorted()Verify that a list of names is kept in alphabetical orderitems.map(x, x.weight).sum() == 1.0Verify that the "weights" of a list of objects sum to 1.0lowPriorities.map(x, x.priority).max() < highPriorities.map(x, x.priority).min()Verify that two sets of priorities do not overlapnames.indexOf('should-be-first') == 1Require that the first name in a list if a specific valueSee the Kubernetes List Library godoc for more information.Kubernetes regex libraryIn addition to the matches function provided by the CEL standard library, the regex library provides find and findAll, enabling a much wider range of regex operations.Examples:Examples of CEL expressions using regex library functionsCEL ExpressionPurpose"abc 123".find('[0-9]+')Find the first number in a string"1, 2, 3, 4".findAll('[0-9]+').map(x, int(x)).sum() < 100Verify that the numbers in a string sum to less than 100See the Kubernetes regex library godoc for more information.Kubernetes URL libraryTo make it easier and safer to process URLs, the following functions have been added:isURL(string) checks if a string is a valid URL according to the Go's net/url package. The string must be an absolute URL.url(string) URL converts a string to a URL or results in an error if the string is not a valid URL.Once parsed via the url function, the resulting URL object has getScheme, getHost, getHostname, getPort, getEscapedPath and getQuery accessors.Examples:Examples of CEL expressions using URL library functionsCEL ExpressionPurposeurl('https://example.com:80/').getHost()Gets the 'example.com:80' host part of the URLurl('https://example.com/path with spaces/').getEscapedPath()Returns '/path%20with%20spaces/'See the Kubernetes URL library godoc for more information.Kubernetes IP address libraryTo make it easier and safer to process IP addresses, the following functions have been added:isIP(string) checks if a string is a valid IP address.ip(string) IP converts a string to an IP address object or results in an error if the string is not a valid IP address.For both functions, the IP address must be an IPv4 or IPv6 address. IPv4-mapped IPv6 addresses (e.g. ::ffff:1.2.3.4) are not allowed. IP addresses with zones (e.g. fe80::1%eth0) are not allowed. Leading zeros in IPv4 address octets are not allowed.Once parsed via the ip function, the resulting IP object has the following library of member functions:Available member functions of an IP address objectMember FunctionCEL Return ValueDescriptionisCanonical()boolReturns true if the IP address is in its canonical form. There is exactly one canonical form for every IP address, so fields containing IPs in canonical form can just be treated as strings when checking for equality or uniqueness.family()intReturns the IP address family, 4 for IPv4 and 6 for IPv6.isUnspecified()boolReturns true if the IP address is the unspecified address. Either the IPv4 address "0.0.0.0" or the IPv6 address "::".isLoopback()boolReturns true if the IP address is the loopback address. Either an IPv4 address with a value of 127.x.x.x or an IPv6 address with a value of ::1.isLinkLocalMulticast()boolReturns true if the IP address is a link-local multicast address. Either an IPv4 address with a value of 224.0.0.x or an IPv6 address in the network ff00::/8.isLinkLocalUnicast()boolReturns true if the IP address is a link-local unicast address. Either an IPv4 address with a value of 169.254.x.x or an IPv6 address in the network fe80::/10.isGlobalUnicast()boolReturns true if the IP address is a global unicast address. Either an IPv4 address that is not zero or 255.255.255.255 or an IPv6 address that is not a link-local unicast, loopback or multicast address.Examples:Examples of CEL expressions using IP address library functionsCEL ExpressionPurposeisIP('127.0.0.1')Returns true for a valid IP.ip('2001:db8::abcd').isCanonical()Returns true for a canonical IPv6.ip('2001:DB8::ABCD').isCanonical()Returns false because the canonical form is lowercase.ip('127.0.0.1').family() == 4Check the address family of an IP.ip('::1').isLoopback()Check if an IP is a loopback address.ip('192.168.0.1').isGlobalUnicast()Check if an IP is a global unicast address.See the Kubernetes IP address library godoc for more information.Kubernetes CIDR libraryCIDR provides a CEL function library extension of CIDR notation parsing functions.cidrConverts a string in CIDR notation to a network address representation or results in an error if the string is not a valid CIDR notation. The CIDR must be an IPv4 or IPv6 subnet address with a mask. Leading zeros in IPv4 address octets are not allowed. IPv4-mapped IPv6 addresses (e.g. ::ffff:1.2.3.4/24) are not allowed.cidr(<string>) <CIDR>Examples:cidr('192.168.0.0/16') // returns an IPv4 address with a CIDR mask cidr('::1/128') // returns an IPv6 address with a CIDR mask cidr('192.168.0.0/33') // error cidr('::1/129') // error cidr('192.168.0.1/16') // error, because there are non-0 bits after the prefixisCIDRReturns true if a string is a valid CIDR notation representation of a subnet with mask. The CIDR must be an IPv4 or IPv6 subnet address with a mask. Leading zeros in IPv4 address octets are not allowed. IPv4-mapped IPv6 addresses (e.g. ::ffff:1.2.3.4/24) are not allowed.isCIDR(<string>) <bool>Examples:isCIDR('192.168.0.0/16') // returns true isCIDR('::1/128') // returns true isCIDR('192.168.0.0/33') // returns false isCIDR('::1/129') // returns falsecontainsIP / containsCIDR / ip / masked / prefixLengthcontainsIP: Returns true if a the CIDR contains the given IP address. The IP address must be an IPv4 or IPv6 address. May take either a string or IP address as an argument.containsCIDR: Returns true if a the CIDR contains the given CIDR. The CIDR must be an IPv4 or IPv6 subnet address with a mask. May take either a string or CIDR as an argument.ip: Returns the IP address representation of the CIDR.masked: Returns the CIDR representation of the network address with a masked prefix. This can be used to return the canonical form of the CIDR network.prefixLength: Returns the prefix length of the CIDR in bits. This is the number of bits in the mask.Examples:Examples of CEL expressions using CIDR library functionsCEL ExpressionPurposecidr('192.168.0.0/24').containsIP(ip('192.168.0.1'))Checks if a CIDR contains a given IP address (IP object).cidr('192.168.0.0/24').containsIP(ip('192.168.1.1'))Checks if a CIDR contains a given IP address (IP object).cidr('192.168.0.0/24').containsIP('192.168.0.1')Checks if a CIDR contains a given IP address (string).cidr('192.168.0.0/24').containsIP('192.168.1.1')Checks if a CIDR contains a given IP address (string).cidr('192.168.0.0/16').containsCIDR(cidr('192.168.10.0/24'))Checks if a CIDR contains another given CIDR (CIDR object).cidr('192.168.1.0/24').containsCIDR(cidr('192.168.2.0/24'))Checks if a CIDR contains another given CIDR (CIDR object).cidr('192.168.0.0/16').containsCIDR('192.168.10.0/24')Checks if a CIDR contains another given CIDR (string).cidr('192.168.1.0/24').containsCIDR('192.168.2.0/24')Checks if a CIDR contains another given CIDR (string).cidr('192.168.0.1/24').ip()Returns the IP address part of a CIDR.cidr('192.168.0.1/24').ip().family()Returns the family of the IP address part of a CIDR.cidr('::1/128').ip()Returns the IP address part of an IPv6 CIDR.cidr('::1/128').ip().family()Returns the family of the IP address part of an IPv6 CIDR.cidr('192.168.0.0/24').masked()Returns the canonical form of a CIDR network.cidr('192.168.0.1/24').masked()Returns the canonical form of a CIDR network, masking non-prefix bits.cidr('192.168.0.0/24') == cidr('192.168.0.0/24').masked()Compares a CIDR to its canonical form (already canonical).cidr('192.168.0.1/24') == cidr('192.168.0.1/24').masked()Compares a CIDR to its canonical form (not canonical).cidr('192.168.0.0/16').prefixLength()Returns the prefix length of an IPv4 CIDR.cidr('::1/128').prefixLength()Returns the prefix length of an IPv6 CIDR.See the Kubernetes CIDR library godoc for more information.Kubernetes authorizer libraryFor CEL expressions in the API where a variable of type Authorizer is available, the authorizer may be used to perform authorization checks for the principal (authenticated user) of the request.API resource checks are performed as follows:Specify the group and resource to check: Authorizer.group(string).resource(string) ResourceCheckOptionally call any combination of the following builder functions to further narrow the authorization check. Note that these functions return the receiver type and can be chained:ResourceCheck.subresource(string) ResourceCheckResourceCheck.namespace(string) ResourceCheckResourceCheck.name(string) ResourceCheckCall ResourceCheck.check(verb string) Decision to perform the authorization check.Call allowed() bool or reason() string to inspect the result of the authorization check.Non-resource authorization performed are used as follows:Specify only a path: Authorizer.path(string) PathCheckCall PathCheck.check(httpVerb string) Decision to perform the authorization check.Call allowed() bool or reason() string to inspect the result of the authorization check.To perform an authorization check for a service account:Authorizer.serviceAccount(namespace string, name string) AuthorizerExamples of CEL expressions using URL library functionsCEL ExpressionPurposeauthorizer.group('').resource('pods').namespace('default').check('create').allowed()Returns true if the principal (user or service account) is allowed create pods in the 'default' namespace.authorizer.path('/healthz').check('get').allowed()Checks if the principal (user or service account) is authorized to make HTTP GET requests to the /healthz API path.authorizer.serviceAccount('default', 'myserviceaccount').resource('deployments').check('delete').allowed()Checks if the service account is authorized to delete deployments.FEATURE STATE: Kubernetes v1.34 [stable] (enabled by default: true)For CEL expressions in the API where a variable of type Authorizer is available, field and label selectors can be included in authorization checks.Examples of CEL expressions using selector authorization functionsCEL ExpressionPurposeauthorizer.group('').resource('pods').fieldSelector('spec.nodeName=mynode').check('list').allowed()Returns true if the principal (user or service account) is allowed to list pods with the field selector spec.nodeName=mynode.authorizer.group('').resource('pods').labelSelector('example.com/mylabel=myvalue').check('list').allowed()Returns true if the principal (user or service account) is allowed to list pods with the label selector example.com/mylabel=myvalue.See the Kubernetes Authz library and Kubernetes AuthzSelectors library godoc for more information.Kubernetes format libraryThe format library provides functions for validating common Kubernetes string formats. This can be useful in the messageExpression of validation rules to provide more specific error messages.The library provides format() functions for each named format, and a generic format.named() function.format.named(string) → ?Format: Returns the Format object for the given format name, if it exists. Otherwise, returns optional.none.format.<formatName>() -> Format: Convenience functions for all the named formats are also available. For example, format.dns1123Label() returns the Format object for DNS-1123 labels.<Format>.validate(string) -> list<string>?: Validates the given string against the format. Returns optional.none if the string is valid, otherwise an optional containing a list of validation error strings.Available Formats:The following format names are supported:Available formats for the format libraryFormat NameDescriptiondns1123LabelValidates if the string is a valid DNS-1123 label.dns1123SubdomainValidates if the string is a valid DNS-1123 subdomain.dns1035LabelValidates if the string is a valid DNS-1035 label.qualifiedNameValidates if the string is a valid qualified name.dns1123LabelPrefixValidates if the string is a valid DNS-1123 label prefix.dns1123SubdomainPrefixValidates if the string is a valid DNS-1123 subdomain prefix.dns1035LabelPrefixValidates if the string is a valid DNS-1035 label prefix.labelValueValidates if the string is a valid label value.uriValidates if the string is a valid URI. Uses the same pattern as `isURL`, but returns an error list.uuidValidates if the string is a valid UUID.byteValidates if the string is a valid base64 encoded string.dateValidates if the string is a valid date in `YYYY-MM-DD` format.datetimeValidates if the string is a valid datetime in RFC3339 format.Examples:Examples of CEL expressions using format library functionsCEL ExpressionPurpose!format.dns1123Label().validate(self.metadata.name).hasValue()A validation rule that checks if an object's name is a valid DNS-1123 label.format.dns1123Label().validate(self.metadata.name).orValue([]).join("\\n")A `messageExpression` that returns specific validation errors for a field. If the field is valid, `validate` returns `optional.none`, and `orValue` provides an empty list, resulting in an empty string.See the Kubernetes Format library godoc for more information.Kubernetes quantity libraryKubernetes 1.28 adds support for manipulating quantity strings (ex 1.5G, 512k, 20Mi)isQuantity(string) checks if a string is a valid Quantity according to Kubernetes' resource.Quantity.quantity(string) Quantity converts a string to a Quantity or results in an error if the string is not a valid quantity.Once parsed via the quantity function, the resulting Quantity object has the following library of member functions:Available member functions of a QuantityMember FunctionCEL Return ValueDescriptionisInteger()boolReturns true if and only if asInteger is safe to call without an errorasInteger()intReturns a representation of the current value as an int64 if possible or results in an error if conversion would result in overflowor loss of precision.asApproximateFloat()floatReturns a float64 representation of the quantity which may lose precision. If the value of the quantity is outside the range of a float64, +Inf/-Inf will be returned.sign()intReturns 1 if the quantity is positive, -1 if it is negative. 0 if it is zero.add(<Quantity>)QuantityReturns sum of two quantitiesadd(<int>)QuantityReturns sum of quantity and an integersub(<Quantity>)QuantityReturns difference between two quantitiessub(<int>)QuantityReturns difference between a quantity and an integerisLessThan(<Quantity>)boolReturns true if and only if the receiver is less than the operandisGreaterThan(<Quantity>)boolReturns true if and only if the receiver is greater than the operandcompareTo(<Quantity>)intCompares receiver to operand and returns 0 if they are equal, 1 if the receiver is greater, or -1 if the receiver is less than the operandExamples:Examples of CEL expressions using URL library functionsCEL ExpressionPurposequantity("500000G").isInteger()Test if conversion to integer would throw an errorquantity("50k").asInteger()Precise conversion to integerquantity("9999999999999999999999999999999999999G").asApproximateFloat()Lossy conversion to floatquantity("50k").add(quantity("20k"))Add two quantitiesquantity("50k").sub(20000)Subtract an integer from a quantityquantity("50k").add(20).sub(quantity("100k")).sub(-50000)Chain adding and subtracting integers and quantitiesquantity("200M").compareTo(quantity("0.2G"))Compare two quantitiesquantity("150Mi").isGreaterThan(quantity("100Mi"))Test if a quantity is greater than the receiverquantity("50M").isLessThan(quantity("100M"))Test if a quantity is less than the receiverKubernetes semver libraryKubernetes v1.34 adds support for parsing and comparing strings that follow the Semantic Versioning 2.0.0 specification. Refer to the semver.org documentation for information on accepted patterns.isSemver(string) checks if a string is a valid semantic version.semver(string) converts a string to a Semver object or results in an error.An optional boolean normalize argument can be passed to isSemver and semver. If true, normalization removes any "v" prefix, adds a 0 minor and patch numbers to versions with only major or major.minor components specified, and removes any leading 0s.Once parsed via the semver function, the resulting Semver object has the following library of member functions:Available member functions of a Semver objectMember FunctionCEL Return ValueDescriptionmajor()intReturns the major version number.minor()intReturns the minor version number.patch()intReturns the patch version number.isLessThan(<Semver>)boolReturns true if and only if the receiver is less than the operand.isGreaterThan(<Semver>)boolReturns true if and only if the receiver is greater than the operand.compareTo(<Semver>)intCompares receiver to operand and returns 0 if they are equal, 1 if the receiver is greater, or -1 if the receiver is less than the operand.Examples:Examples of CEL expressions using semver library functionsCEL ExpressionPurposeisSemver('1.0.0')Returns true for a valid Semver string.isSemver('v1.0', true)Returns true for a normalizable Semver string.semver('1.2.3').major()Returns the major version of a Semver.semver('1.2.3').compareTo(semver('2.0.0')) < 0Compare two Semver strings.See the Kubernetes Semver library godoc for more information.Type checkingCEL is a gradually typed language.Some Kubernetes API fields contain fully type checked CEL expressions. For example, CustomResourceDefinitions Validation Rules are fully type checked.Some Kubernetes API fields contain partially type checked CEL expressions. A partially type checked expression is an expressions where some of the variables are statically typed but others are dynamically typed. For example, in the CEL expressions of ValidatingAdmissionPolicies the request variable is typed, but the object variable is dynamically typed. As a result, an expression containing request.namex would fail type checking because the namex field is not defined. However, object.namex would pass type checking even when the namex field is not defined for the resource kinds that object refers to, because object is dynamically typed.The has() macro in CEL may be used in CEL expressions to check if a field of a dynamically typed variable is accessible before attempting to access the field's value. For example:has(object.namex) ? object.namex == 'special' : request.name == 'special' Type system integrationTable showing the relationship between OpenAPIv3 types and CEL typesOpenAPIv3 typeCEL type'object' with Propertiesobject / "message type" (type(<object>) evaluates to selfType<uniqueNumber>.path.to.object.from.self)'object' with additionalPropertiesmap'object' with x-kubernetes-embedded-typeobject / "message type", 'apiVersion', 'kind', 'metadata.name' and 'metadata.generateName' are implicitly included in schema'object' with x-kubernetes-preserve-unknown-fieldsobject / "message type", unknown fields are NOT accessible in CEL expressionx-kubernetes-int-or-stringUnion of int or string, self.intOrString < 100 | self.intOrString == '50%' evaluates to true for both 50 and "50%"'array'list'array' with x-kubernetes-list-type=maplist with map based Equality & unique key guarantees'array' with x-kubernetes-list-type=setlist with set based Equality & unique entry guarantees'boolean'boolean'number' (all formats)double'integer' (all formats)int (64)no equivalentuint (64)'null'null_type'string'string'string' with format=byte (base64 encoded)bytes'string' with format=datetimestamp (google.protobuf.Timestamp)'string' with format=datetimetimestamp (google.protobuf.Timestamp)'string' with format=durationduration (google.protobuf.Duration)Also see: CEL types, OpenAPI types, Kubernetes Structural Schemas.Equality comparison for arrays with x-kubernetes-list-type of set or map ignores element order. For example [1, 2] == [2, 1] if the arrays represent Kubernetes set values.Concatenation on arrays with x-kubernetes-list-type use the semantics of the list type:setX + Y performs a union where the array positions of all elements in X are preserved and non-intersecting elements in Y are appended, retaining their partial order.mapX + Y performs a merge where the array positions of all keys in X are preserved but the values are overwritten by values in Y when the key sets of X and Y intersect. Elements in Y with non-intersecting keys are appended, retaining their partial order.EscapingOnly Kubernetes resource property names of the form [a-zA-Z_.-/][a-zA-Z0-9_.-/]* are accessible from CEL. Accessible property names are escaped according to the following rules when accessed in the expression:Table of CEL identifier escaping rulesescape sequenceproperty name equivalent__underscores______dot__.__dash__-__slash__/__{keyword}__CEL RESERVED keywordWhen you escape any of CEL's RESERVED keywords you need to match the exact property name use the underscore escaping (for example, int in the word sprint would not be escaped and nor would it need to be).Examples on escaping:Examples escaped CEL identifiersproperty namerule with escaped property namenamespaceself.__namespace__ > 0x-propself.x__dash__prop > 0redact_dself.redact__underscores__d > 0stringself.startsWith('kube')Resource constraintsCEL is non-Turing complete and offers a variety of production safety controls to limit execution time. CEL's resource constraint features provide


SelfSubjectRulesReviewSelfSubjectRulesReview enumerates the set of actions the current user can perform within a namespace.apiVersion: authorization.k8s.io/v1import "k8s.io/api/authorization/v1"SelfSubjectRulesReviewSelfSubjectRulesReview enumerates the set of actions the current user can perform within a namespace. The returned list of actions may be incomplete depending on the server's authorization mode, and any errors experienced during the evaluation. SelfSubjectRulesReview should be used by UIs to show/hide actions, or to quickly let an end user reason about their permissions. It should NOT Be used by external systems to drive authorization decisions as this raises confused deputy, cache lifetime/revocation, and correctness concerns. SubjectAccessReview, and LocalAccessReview are the correct way to defer authorization decisions to the API server.apiVersion: authorization.k8s.io/v1kind: SelfSubjectRulesReviewmetadata (ObjectMeta)Standard list metadata. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadataspec (SelfSubjectRulesReviewSpec), requiredSpec holds information about the request being evaluated.status (SubjectRulesReviewStatus)Status is filled in by the server and indicates the set of actions a user can perform.SubjectRulesReviewStatus contains the result of a rules check. This check can be incomplete depending on the set of authorizers the server is configured with and any errors experienced during evaluation. Because authorization rules are additive, if a rule appears in a list it's safe to assume the subject has that permission, even if that list is incomplete.status.incomplete (boolean), requiredIncomplete is true when the rules returned by this call are incomplete. This is most commonly encountered when an authorizer, such as an external authorizer, doesn't support rules evaluation.status.nonResourceRules ([]NonResourceRule), requiredAtomic: will be replaced during a mergeNonResourceRules is the list of actions the subject is allowed to perform on non-resources. The list ordering isn't significant, may contain duplicates, and possibly be incomplete.NonResourceRule holds information that describes a rule for the non-resourcestatus.nonResourceRules.verbs ([]string), requiredAtomic: will be replaced during a mergeVerb is a list of kubernetes non-resource API verbs, like: get, post, put, delete, patch, head, options. "*" means all.status.nonResourceRules.nonResourceURLs ([]string)Atomic: will be replaced during a mergeNonResourceURLs is a set of partial urls that a user should have access to. s are allowed, but only as the full, final step in the path. "" means all.status.resourceRules ([]ResourceRule), requiredAtomic: will be replaced during a mergeResourceRules is the list of actions the subject is allowed to perform on resources. The list ordering isn't significant, may contain duplicates, and possibly be incomplete.ResourceRule is the list of actions the subject is allowed to perform on resources. The list ordering isn't significant, may contain duplicates, and possibly be incomplete.status.resourceRules.verbs ([]string), requiredAtomic: will be replaced during a mergeVerb is a list of kubernetes resource API verbs, like: get, list, watch, create, update, delete, proxy. "*" means all.status.resourceRules.apiGroups ([]string)Atomic: will be replaced during a mergeAPIGroups is the name of the APIGroup that contains the resources. If multiple API groups are specified, any action requested against one of the enumerated resources in any API group will be allowed. "*" means all.status.resourceRules.resourceNames ([]string)Atomic: will be replaced during a mergeResourceNames is an optional white list of names that the rule applies to. An empty set means that everything is allowed. "*" means all.status.resourceRules.resources ([]string)Atomic: will be replaced during a mergeResources is a list of resources this rule applies to. "" means all in the specified apiGroups. "/foo" represents the subresource 'foo' for all resources in the specified apiGroups.status.evaluationError (string)EvaluationError can appear in combination with Rules. It indicates an error occurred during rule evaluation, such as an authorizer that doesn't support rule evaluation, and that ResourceRules and/or NonResourceRules may be incomplete.SelfSubjectRulesReviewSpecSelfSubjectRulesReviewSpec defines the specification for SelfSubjectRulesReview.namespace (string)Namespace to evaluate rules for. Required.Operationscreate create a SelfSubjectRulesReviewHTTP RequestPOST /apis/authorization.k8s.io/v1/selfsubjectrulesreviewsParametersbody: SelfSubjectRulesReview, requireddryRun (in query): stringdryRunfieldManager (in query): stringfieldManagerfieldValidation (in query): stringfieldValidationpretty (in query): stringprettyResponse200 (SelfSubjectRulesReview): OK201 (SelfSubjectRulesReview): Created202 (SelfSubjectRulesReview): Accepted401: UnauthorizedThis page is automatically generated.If you plan to report an issue with this page, mention that the page is auto-generated in your issue description. The fix may need to happen elsewhere in the Kubernetes project.

SelfSubjectRulesReview


kubectl create namespaceSynopsisCreate a namespace with the specified name.kubectl create namespace NAME [--dry-run=server|client|none] Examples # Create a new namespace named my-namespace kubectl create namespace my-namespace Options--allow-missing-template-keys Default: trueIf true, ignore any errors in templates when a field or map key is missing in the template. Only applies to golang and jsonpath output formats.--dry-run string[="unchanged"] Default: "none"Must be "none", "server", or "client". If client strategy, only print the object that would be sent, without sending it. If server strategy, submit server-side request without persisting the resource.--field-manager string Default: "kubectl-create"Name of the manager used to track field ownership.-h, --helphelp for namespace-o, --output stringOutput format. One of: (json, yaml, name, go-template, go-template-file, template, templatefile, jsonpath, jsonpath-as-json, jsonpath-file).--save-configIf true, the configuration of current object will be saved in its annotation. Otherwise, the annotation will be unchanged. This flag is useful when you want to perform kubectl apply on this object in the future.--show-managed-fieldsIf true, keep the managedFields when printing objects in JSON or YAML format.--template stringTemplate string or path to template file to use when -o=go-template, -o=go-template-file. The template format is golang templates [http://golang.org/pkg/text/template/#pkg-overview].--validate string[="strict"] Default: "strict"Must be one of: strict (or true), warn, ignore (or false). "true" or "strict" will use a schema to validate the input and fail the request if invalid. It will perform server side validation if ServerSideFieldValidation is enabled on the api-server, but will fall back to less reliable client-side validation if not. "warn" will warn about unknown or duplicate fields without blocking the request if server-side field validation is enabled on the API server, and behave as "ignore" otherwise. "false" or "ignore" will not perform any schema validation, silently dropping any unknown or duplicate fields.Parent Options Inherited--as stringUsername to impersonate for the operation. User could be a regular user or a service account in a namespace.--as-group stringsGroup to impersonate for the operation, this flag can be repeated to specify multiple groups.--as-uid stringUID to impersonate for the operation.--cache-dir string Default: "$HOME/.kube/cache"Default cache directory--certificate-authority stringPath to a cert file for the certificate authority--client-certificate stringPath to a client certificate file for TLS--client-key stringPath to a client key file for TLS--cluster stringThe name of the kubeconfig cluster to use--context stringThe name of the kubeconfig context to use--disable-compressionIf true, opt-out of response compression for all requests to the server--insecure-skip-tls-verifyIf true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure--kubeconfig stringPath to the kubeconfig file to use for CLI requests.--kuberc stringPath to the kuberc file to use for preferences. This can be disabled by exporting KUBECTL_KUBERC=false feature gate or turning off the feature KUBERC=off.--match-server-versionRequire server version to match client version-n, --namespace stringIf present, the namespace scope for this CLI request--password stringPassword for basic authentication to the API server--profile string Default: "none"Name of profile to capture. One of (none|cpu|heap|goroutine|threadcreate|block|mutex)--profile-output string Default: "profile.pprof"Name of the file to write the profile to--request-timeout string Default: "0"The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests.-s, --server stringThe address and port of the Kubernetes API server--storage-driver-buffer-duration duration Default: 1m0sWrites in the storage driver will be buffered for this duration, and committed to the non memory backends as a single transaction--storage-driver-db string Default: "cadvisor"database name--storage-driver-host string Default: "localhost:8086"database host:port--storage-driver-password string Default: "root"database password--storage-driver-secureuse secure connection with database--storage-driver-table string Default: "stats"table name--storage-driver-user string Default: "root"database username--tls-server-name stringServer name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used--token stringBearer token for authentication to the API server--user stringThe name of the kubeconfig user to use--username stringUsername for basic authentication to the API server--version version[=true]--version, --version=raw prints version information and quits; --version=vX.Y.Z... sets the reported version--warnings-as-errorsTreat warnings received from the server as errors and exit with a non-zero exit codeSee Alsokubectl create - Create a resource from a file or from stdinThis page is automatically generated.If you plan to report an issue with this page, mention that the page is auto-generated in your issue description. The fix may need to happen elsewhere in the Kubernetes project.

kubectl create namespace


Install and Set Up kubectl on LinuxBefore you beginYou must use a kubectl version that is within one minor version difference of your cluster. For example, a v1.34 client can communicate with v1.33, v1.34, and v1.35 control planes. Using the latest compatible version of kubectl helps avoid unforeseen issues.Install kubectl on LinuxThe following methods exist for installing kubectl on Linux:Install kubectl binary with curl on LinuxInstall using native package managementInstall using other package managementInstall kubectl binary with curl on LinuxDownload the latest release with the command: curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl" curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/arm64/kubectl" Note:To download a specific version, replace the $(curl -L -s https://dl.k8s.io/release/stable.txt) portion of the command with the specific version.For example, to download version 1.34.0 on Linux x86-64, type:curl -LO https://dl.k8s.io/release/v1.34.0/bin/linux/amd64/kubectl And for Linux ARM64, type:curl -LO https://dl.k8s.io/release/v1.34.0/bin/linux/arm64/kubectl Validate the binary (optional)Download the kubectl checksum file: curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl.sha256" curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/arm64/kubectl.sha256" Validate the kubectl binary against the checksum file:echo "$(cat kubectl.sha256) kubectl" | sha256sum --check If valid, the output is:kubectl: OK If the check fails, sha256 exits with nonzero status and prints output similar to:kubectl: FAILED sha256sum: WARNING: 1 computed checksum did NOT match Note:Download the same version of the binary and checksum.Install kubectlsudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl Note:If you do not have root access on the target system, you can still install kubectl to the ~/.local/bin directory:chmod +x kubectl mkdir -p ~/.local/bin mv ./kubectl ~/.local/bin/kubectl # and then append (or prepend) ~/.local/bin to $PATH Test to ensure the version you installed is up-to-date:kubectl version --client Or use this for detailed view of version:kubectl version --client --output=yaml Install using native package managementUpdate the apt package index and install packages needed to use the Kubernetes apt repository:sudo apt-get update # apt-transport-https may be a dummy package; if so, you can skip that package sudo apt-get install -y apt-transport-https ca-certificates curl gnupg Download the public signing key for the Kubernetes package repositories. The same signing key is used for all repositories so you can disregard the version in the URL:# If the folder `/etc/apt/keyrings` does not exist, it should be created before the curl command, read the note below. # sudo mkdir -p -m 755 /etc/apt/keyrings curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.34/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg sudo chmod 644 /etc/apt/keyrings/kubernetes-apt-keyring.gpg # allow unprivileged APT programs to read this keyring Note:In releases older than Debian 12 and Ubuntu 22.04, folder /etc/apt/keyrings does not exist by default, and it should be created before the curl command.Add the appropriate Kubernetes apt repository. If you want to use Kubernetes version different than v1.34, replace v1.34 with the desired minor version in the command below:# This overwrites any existing configuration in /etc/apt/sources.list.d/kubernetes.list echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.34/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list sudo chmod 644 /etc/apt/sources.list.d/kubernetes.list # helps tools such as command-not-found to work correctly Note:To upgrade kubectl to another minor release, you'll need to bump the version in /etc/apt/sources.list.d/kubernetes.list before running apt-get update and apt-get upgrade. This procedure is described in more detail in Changing The Kubernetes Package Repository.Update apt package index, then install kubectl:sudo apt-get update sudo apt-get install -y kubectl Add the Kubernetes yum repository. If you want to use Kubernetes version different than v1.34, replace v1.34 with the desired minor version in the command below.# This overwrites any existing configuration in /etc/yum.repos.d/kubernetes.repo cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo [kubernetes] name=Kubernetes baseurl=https://pkgs.k8s.io/core:/stable:/v1.34/rpm/ enabled=1 gpgcheck=1 gpgkey=https://pkgs.k8s.io/core:/stable:/v1.34/rpm/repodata/repomd.xml.key EOF Note:To upgrade kubectl to another minor release, you'll need to bump the version in /etc/yum.repos.d/kubernetes.repo before running yum update. This procedure is described in more detail in Changing The Kubernetes Package Repository.Install kubectl using yum:sudo yum install -y kubectl Add the Kubernetes zypper repository. If you want to use Kubernetes version different than v1.34, replace v1.34 with the desired minor version in the command below.# This overwrites any existing configuration in /etc/zypp/repos.d/kubernetes.repo cat <<EOF | sudo tee /etc/zypp/repos.d/kubernetes.repo [kubernetes] name=Kubernetes baseurl=https://pkgs.k8s.io/core:/stable:/v1.34/rpm/ enabled=1 gpgcheck=1 gpgkey=https://pkgs.k8s.io/core:/stable:/v1.34/rpm/repodata/repomd.xml.key EOF Note:To upgrade kubectl to another minor release, you'll need to bump the version in /etc/zypp/repos.d/kubernetes.repo before running zypper update. This procedure is described in more detail in Changing The Kubernetes Package Repository.Update zypper and confirm the new repo addition:sudo zypper update When this message appears, press 't' or 'a':New repository or package signing key received: Repository: Kubernetes Key Fingerprint: 1111 2222 3333 4444 5555 6666 7777 8888 9999 AAAA Key Name: isv:kubernetes OBS Project <isv:kubernetes@build.opensuse.org> Key Algorithm: RSA 2048 Key Created: Thu 25 Aug 2022 01:21:11 PM -03 Key Expires: Sat 02 Nov 2024 01:21:11 PM -03 (expires in 85 days) Rpm Name: gpg-pubkey-9a296436-6307a177 Note: Signing data enables the recipient to verify that no modifications occurred after the data were signed. Accepting data with no, wrong or unknown signature can lead to a corrupted system and in extreme cases even to a system compromise. Note: A GPG pubkey is clearly identified by its fingerprint. Do not rely on the key's name. If you are not sure whether the presented key is authentic, ask the repository provider or check their web site. Many providers maintain a web page showing the fingerprints of the GPG keys they are using. Do you want to reject the key, trust temporarily, or trust always? [r/t/a/?] (r): a Install kubectl using zypper:sudo zypper install -y kubectl Install using other package managementIf you are on Ubuntu or another Linux distribution that supports the snap package manager, kubectl is available as a snap application.snap install kubectl --classic kubectl version --client If you are on Linux and using Homebrew package manager, kubectl is available for installation.brew install kubectl kubectl version --client Verify kubectl configurationIn order for kubectl to find and access a Kubernetes cluster, it needs a kubeconfig file, which is created automatically when you create a cluster using kube-up.sh or successfully deploy a Minikube cluster. By default, kubectl configuration is located at ~/.kube/config.Check that kubectl is properly configured by getting the cluster state:kubectl cluster-info If you see a URL response, kubectl is correctly configured to access your cluster.If you see a message similar to the following, kubectl is not configured correctly or is not able to connect to a Kubernetes cluster.The connection to the server <server-name:port> was refused - did you specify the right host or port? For example, if you are intending to run a Kubernetes cluster on your laptop (locally), you will need a tool like Minikube to be installed first and then re-run the commands stated above.If kubectl cluster-info returns the url response but you can't access your cluster, to check whether it is configured properly, use:kubectl cluster-info dump Troubleshooting the 'No Auth Provider Found' error messageIn Kubernetes 1.26, kubectl removed the built-in authentication for the following cloud providers' managed Kubernetes offerings. These providers have released kubectl plugins to provide the cloud-specific authentication. For instructions, refer to the following provider documentation:Azure AKS: kubelogin pluginGoogle Kubernetes Engine: gke-gcloud-auth-plugin(There could also be other reasons to see the same error message, unrelated to that change.)Optional kubectl configurations and pluginsEnable shell autocompletionkubectl provides autocompletion support for Bash, Zsh, Fish, and PowerShell, which can save you a lot of typing.Below are the procedures to set up autocompletion for Bash, Fish, and Zsh.IntroductionThe kubectl completion script for Bash can be generated with the command kubectl completion bash. Sourcing the completion script in your shell enables kubectl autocompletion.However, the completion script depends on bash-completion, which means that you have to install this software first (you can test if you have bash-completion already installed by running type _init_completion).Install bash-completionbash-completion is provided by many package managers (see here). You can install it with apt-get install bash-completion or yum install bash-completion, etc.The above commands create /usr/share/bash-completion/bash_completion, which is the main script of bash-completion. Depending on your package manager, you have to manually source this file in your ~/.bashrc file.To find out, reload your shell and run type _init_completion. If the command succeeds, you're already set, otherwise add the following to your ~/.bashrc file:source /usr/share/bash-completion/bash_completion Reload your shell and verify that bash-completion is correctly installed by typing type _init_completion.Enable kubectl autocompletionBashYou now need to ensure that the kubectl completion script gets sourced in all your shell sessions. There are two ways in which you can do this: echo 'source <(kubectl completion bash)' >>~/.bashrc kubectl completion bash | sudo tee /etc/bash_completion.d/kubectl > /dev/null sudo chmod a+r /etc/bash_completion.d/kubectl If you have an alias for kubectl, you can extend shell completion to work with that alias:echo 'alias k=kubectl' >>~/.bashrc echo 'complete -o default -F __start_kubectl k' >>~/.bashrc Note:bash-completion sources all completion scripts in /etc/bash_completion.d.Both approaches are equivalent. After reloading your shell, kubectl autocompletion should be working. To enable bash autocompletion in current session of shell, source the ~/.bashrc file:source ~/.bashrc Note:Autocomplete for Fish requires kubectl 1.23 or later.The kubectl completion script for Fish can be generated with the command kubectl completion fish. Sourcing the completion script in your shell enables kubectl autocompletion.To do so in all your shell sessions, add the following line to your ~/.config/fish/config.fish file:kubectl completion fish | source After reloading your shell, kubectl autocompletion should be working.The kubectl completion script for Zsh can be generated with the command kubectl completion zsh. Sourcing the completion script in your shell enables kubectl autocompletion.To do so in all your shell sessions, add the following to your ~/.zshrc file:source <(kubectl completion zsh) If you have an alias for kubectl, kubectl autocompletion will automatically work with it.After reloading your shell, kubectl autocompletion should be working.If you get an error like 2: command not found: compdef, then add the following to the beginning of your ~/.zshrc file:autoload -Uz compinit compinit Configure kubercSee kuberc for more information.Install kubectl convert pluginA plugin for Kubernetes command-line tool kubectl, which allows you to convert manifests between different API versions. This can be particularly helpful to migrate manifests to a non-deprecated api version with newer Kubernetes release. For more info, visit migrate to non deprecated apisDownload the latest release with the command: curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl-convert" curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/arm64/kubectl-convert" Validate the binary (optional)Download the kubectl-convert checksum file: curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl-convert.sha256" curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/arm64/kubectl-convert.sha256" Validate the kubectl-convert binary against the checksum file:echo "$(cat kubectl-convert.sha256) kubectl-convert" | sha256sum --check If valid, the output is:kubectl-convert: OK If the check fails, sha256 exits with nonzero status and prints output similar to:kubectl-convert: FAILED sha256sum: WARNING: 1 computed checksum did NOT match Note:Download the same version of the binary and checksum.Install kubectl-convertsudo install -o root -g root -m 0755 kubectl-convert /usr/local/bin/kubectl-convert Verify plugin is successfully installedkubectl convert --help If you do not see an error, it means the plugin is successfully installed.After installing the plugin, clean up the installation files:rm kubectl-convert kubectl-convert.sha256 What's nextInstall MinikubeSee the getting started guides for more about creating clusters.Learn how to launch and expose your application.If you need access to a cluster you didn't create, see the Sharing Cluster Access document.Read the kubectl reference docs

Install and Set Up kubectl on Linux


Configure Minimum and Maximum Memory Constraints for a NamespaceDefine a range of valid memory resource limits for a namespace, so that every new Pod in that namespace falls within the range you configure.This page shows how to set minimum and maximum values for memory used by containers running in a namespace. You specify minimum and maximum memory values in a LimitRange object. If a Pod does not meet the constraints imposed by the LimitRange, it cannot be created in the namespace.Before you beginYou need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. It is recommended to run this tutorial on a cluster with at least two nodes that are not acting as control plane hosts. If you do not already have a cluster, you can create one by using minikube or you can use one of these Kubernetes playgrounds:iximiuz LabsKillercodaKodeKloudPlay with KubernetesYou must have access to create namespaces in your cluster.Each node in your cluster must have at least 1 GiB of memory available for Pods.Create a namespaceCreate a namespace so that the resources you create in this exercise are isolated from the rest of your cluster.kubectl create namespace constraints-mem-example Create a LimitRange and a PodHere's an example manifest for a LimitRange:admin/resource/memory-constraints.yaml apiVersion: v1 kind: LimitRange metadata: name: mem-min-max-demo-lr spec: limits: - max: memory: 1Gi min: memory: 500Mi type: Container Create the LimitRange:kubectl apply -f https://k8s.io/examples/admin/resource/memory-constraints.yaml --namespace=constraints-mem-example View detailed information about the LimitRange:kubectl get limitrange mem-min-max-demo-lr --namespace=constraints-mem-example --output=yaml The output shows the minimum and maximum memory constraints as expected. But notice that even though you didn't specify default values in the configuration file for the LimitRange, they were created automatically. limits: - default: memory: 1Gi defaultRequest: memory: 1Gi max: memory: 1Gi min: memory: 500Mi type: Container Now whenever you define a Pod within the constraints-mem-example namespace, Kubernetes performs these steps:If any container in that Pod does not specify its own memory request and limit, the control plane assigns the default memory request and limit to that container.Verify that every container in that Pod requests at least 500 MiB of memory.Verify that every container in that Pod requests no more than 1024 MiB (1 GiB) of memory.Here's a manifest for a Pod that has one container. Within the Pod spec, the sole container specifies a memory request of 600 MiB and a memory limit of 800 MiB. These satisfy the minimum and maximum memory constraints imposed by the LimitRange.admin/resource/memory-constraints-pod.yaml apiVersion: v1 kind: Pod metadata: name: constraints-mem-demo spec: containers: - name: constraints-mem-demo-ctr image: nginx resources: limits: memory: "800Mi" requests: memory: "600Mi" Create the Pod:kubectl apply -f https://k8s.io/examples/admin/resource/memory-constraints-pod.yaml --namespace=constraints-mem-example Verify that the Pod is running and that its container is healthy:kubectl get pod constraints-mem-demo --namespace=constraints-mem-example View detailed information about the Pod:kubectl get pod constraints-mem-demo --output=yaml --namespace=constraints-mem-example The output shows that the container within that Pod has a memory request of 600 MiB and a memory limit of 800 MiB. These satisfy the constraints imposed by the LimitRange for this namespace:resources: limits: memory: 800Mi requests: memory: 600Mi Delete your Pod:kubectl delete pod constraints-mem-demo --namespace=constraints-mem-example Attempt to create a Pod that exceeds the maximum memory constraintHere's a manifest for a Pod that has one container. The container specifies a memory request of 800 MiB and a memory limit of 1.5 GiB.admin/resource/memory-constraints-pod-2.yaml apiVersion: v1 kind: Pod metadata: name: constraints-mem-demo-2 spec: containers: - name: constraints-mem-demo-2-ctr image: nginx resources: limits: memory: "1.5Gi" requests: memory: "800Mi" Attempt to create the Pod:kubectl apply -f https://k8s.io/examples/admin/resource/memory-constraints-pod-2.yaml --namespace=constraints-mem-example The output shows that the Pod does not get created, because it defines a container that requests more memory than is allowed:Error from server (Forbidden): error when creating "examples/admin/resource/memory-constraints-pod-2.yaml": pods "constraints-mem-demo-2" is forbidden: maximum memory usage per Container is 1Gi, but limit is 1536Mi. Attempt to create a Pod that does not meet the minimum memory requestHere's a manifest for a Pod that has one container. That container specifies a memory request of 100 MiB and a memory limit of 800 MiB.admin/resource/memory-constraints-pod-3.yaml apiVersion: v1 kind: Pod metadata: name: constraints-mem-demo-3 spec: containers: - name: constraints-mem-demo-3-ctr image: nginx resources: limits: memory: "800Mi" requests: memory: "100Mi" Attempt to create the Pod:kubectl apply -f https://k8s.io/examples/admin/resource/memory-constraints-pod-3.yaml --namespace=constraints-mem-example The output shows that the Pod does not get created, because it defines a container that requests less memory than the enforced minimum:Error from server (Forbidden): error when creating "examples/admin/resource/memory-constraints-pod-3.yaml": pods "constraints-mem-demo-3" is forbidden: minimum memory usage per Container is 500Mi, but request is 100Mi. Create a Pod that does not specify any memory request or limitHere's a manifest for a Pod that has one container. The container does not specify a memory request, and it does not specify a memory limit.admin/resource/memory-constraints-pod-4.yaml apiVersion: v1 kind: Pod metadata: name: constraints-mem-demo-4 spec: containers: - name: constraints-mem-demo-4-ctr image: nginx Create the Pod:kubectl apply -f https://k8s.io/examples/admin/resource/memory-constraints-pod-4.yaml --namespace=constraints-mem-example View detailed information about the Pod:kubectl get pod constraints-mem-demo-4 --namespace=constraints-mem-example --output=yaml The output shows that the Pod's only container has a memory request of 1 GiB and a memory limit of 1 GiB. How did that container get those values?resources: limits: memory: 1Gi requests: memory: 1Gi Because your Pod did not define any memory request and limit for that container, the cluster applied a default memory request and limit from the LimitRange.This means that the definition of that Pod shows those values. You can check it using kubectl describe:# Look for the "Requests:" section of the output kubectl describe pod constraints-mem-demo-4 --namespace=constraints-mem-example At this point, your Pod might be running or it might not be running. Recall that a prerequisite for this task is that your Nodes have at least 1 GiB of memory. If each of your Nodes has only 1 GiB of memory, then there is not enough allocatable memory on any Node to accommodate a memory request of 1 GiB. If you happen to be using Nodes with 2 GiB of memory, then you probably have enough space to accommodate the 1 GiB request.Delete your Pod:kubectl delete pod constraints-mem-demo-4 --namespace=constraints-mem-example Enforcement of minimum and maximum memory constraintsThe maximum and minimum memory constraints imposed on a namespace by a LimitRange are enforced only when a Pod is created or updated. If you change the LimitRange, it does not affect Pods that were created previously.Motivation for minimum and maximum memory constraintsAs a cluster administrator, you might want to impose restrictions on the amount of memory that Pods can use. For example:Each Node in a cluster has 2 GiB of memory. You do not want to accept any Pod that requests more than 2 GiB of memory, because no Node in the cluster can support the request.A cluster is shared by your production and development departments. You want to allow production workloads to consume up to 8 GiB of memory, but you want development workloads to be limited to 512 MiB. You create separate namespaces for production and development, and you apply memory constraints to each namespace.Clean upDelete your namespace:kubectl delete namespace constraints-mem-example What's nextFor cluster administratorsConfigure Default Memory Requests and Limits for a NamespaceConfigure Default CPU Requests and Limits for a NamespaceConfigure Minimum and Maximum CPU Constraints for a NamespaceConfigure Memory and CPU Quotas for a NamespaceConfigure a Pod Quota for a NamespaceConfigure Quotas for API ObjectsFor app developersAssign Memory Resources to Containers and PodsAssign CPU Resources to Containers and PodsAssign Pod-level CPU and memory resourcesConfigure Quality of Service for Pods