Reviewing pull requestsAnyone can review a documentation pull request. Visit the pull requests section in the Kubernetes website repository to see open pull requests.Reviewing documentation pull requests is a great way to introduce yourself to the Kubernetes community. It helps you learn the code base and build trust with other contributors.Before reviewing, it's a good idea to:Read the content guide and style guide so you can leave informed comments.Understand the different roles and responsibilities in the Kubernetes documentation community.Before you beginBefore you start a review:Read the CNCF Code of Conduct and ensure that you abide by it at all times.Be polite, considerate, and helpful.Comment on positive aspects of PRs as well as changes.Be empathetic and mindful of how your review may be received.Assume good intent and ask clarifying questions.Experienced contributors, consider pairing with new contributors whose work requires extensive changes.Review processIn general, review pull requests for content and style in English. Figure 1 outlines the steps for the review process. The details for each step follow.flowchart LR subgraph fourth[Start review] direction TB S[ ] -.- M[add comments] --> N[review changes] N --> O[new contributors shouldchoose Comment] end subgraph third[Select PR] direction TB T[ ] -.- J[read descriptionand comments]--> K[preview changes inNetlify preview build] end A[Review open PR list]--> B[Filter open PRsby label] B --> third --> fourth classDef grey fill:#dddddd,stroke:#ffffff,stroke-width:px,color:#000000, font-size:15px; classDef white fill:#ffffff,stroke:#000,stroke-width:px,color:#000,font-weight:bold classDef spacewhite fill:#ffffff,stroke:#fff,stroke-width:0px,color:#000 class A,B,J,K,M,N,O grey class S,T spacewhite class third,fourth whiteJavaScript must be enabled to view this contentFigure 1. Review process steps.Go to https://github.com/kubernetes/website/pulls. You see a list of every open pull request against the Kubernetes website and docs.Filter the open PRs using one or all of the following labels:cncf-cla: yes (Recommended): PRs submitted by contributors who have not signed the CLA cannot be merged. See Sign the CLA for more information.language/en (Recommended): Filters for english language PRs only.size/<size>: filters for PRs of a certain size. If you're new, start with smaller PRs.Additionally, ensure the PR isn't marked as a work in progress. PRs using the work in progress label are not ready for review yet.Once you've selected a PR to review, understand the change by:Reading the PR description to understand the changes made, and read any linked issuesReading any comments by other reviewersClicking the Files changed tab to see the files and lines changedPreviewing the changes in the Netlify preview build by scrolling to the PR's build check section at the bottom of the Conversation tab. Here's a screenshot (this shows GitHub's desktop site; if you're reviewing on a tablet or smartphone device, the GitHub web UI is slightly different):To open the preview, click on the Details link of the deploy/netlify line in the list of checks.Go to the Files changed tab to start your review.Click on the + symbol beside the line you want to comment on.Fill in any comments you have about the line and click either Add single comment (if you have only one comment to make) or Start a review (if you have multiple comments to make).When finished, click Review changes at the top of the page. Here, you can add a summary of your review (and leave some positive comments for the contributor!). Please always use the "Comment"Avoid clicking the "Request changes" button when finishing your review. If you want to block a PR from being merged before some further changes are made, you can leave a "/hold" comment. Mention why you are setting a hold, and optionally specify the conditions under which the hold can be removed by you or other reviewers.Avoid clicking the "Approve" button when finishing your review. Leaving a "/approve" comment is recommended most of the time.Reviewing checklistWhen reviewing, use the following as a starting point.Language and grammarAre there any obvious errors in language or grammar? Is there a better way to phrase something?Focus on the language and grammar of the parts of the page that the author is changing. Unless the author is clearly aiming to update the entire page, they have no obligation to fix every issue on the page.When a PR updates an existing page, you should focus on reviewing the parts of the page that are being updated. That changed content should be reviewed for technical and editorial correctness. If you find errors on the page that don't directly relate to what the PR author is attempting to address, then it should be treated as a separate issue (check that there isn't an existing issue about this first).Watch out for pull requests that move content. If an author renames a page or combines two pages, we (Kubernetes SIG Docs) usually avoid asking that author to fix every grammar or spelling nit that we could spot within that moved content.Are there any complicated or archaic words which could be replaced with a simpler word?Are there any words, terms or phrases in use which could be replaced with a non-discriminatory alternative?Does the word choice and its capitalization follow the style guide?Are there long sentences which could be shorter or less complex?Are there any long paragraphs which might work better as a list or table?ContentDoes similar content exist elsewhere on the Kubernetes site?Does the content excessively link to off-site, individual vendor or non-open source documentation?DocumentationSome checks to consider:Did this PR change or remove a page title, slug/alias or anchor link? If so, are there broken links as a result of this PR? Is there another option, like changing the page title without changing the slug?Does the PR introduce a new page? If so:Is the page using the right page content type and associated Hugo shortcodes?Does the page appear correctly in the section's side navigation (or at all)?Should the page appear on the Docs Home listing?Do the changes show up in the Netlify preview? Be particularly vigilant about lists, code blocks, tables, notes and images.BlogEarly

Reviewing pull requests


Resource QuotasWhen several users or teams share a cluster with a fixed number of nodes, there is a concern that one team could use more than its fair share of resources.Resource quotas are a tool for administrators to address this concern.A resource quota, defined by a ResourceQuota object, provides constraints that limit aggregate resource consumption per namespace. A ResourceQuota can also limit the quantity of objects that can be created in a namespace by API kind, as well as the total amount of infrastructure resources that may be consumed by API objects found in that namespace.Caution:Neither contention nor changes to quota will affect already created resources.How Kubernetes ResourceQuotas workResourceQuotas work like this:Different teams work in different namespaces. This separation can be enforced with RBAC or any other authorization mechanism.A cluster administrator creates at least one ResourceQuota for each namespace.To make sure the enforcement stays enforced, the cluster administrator should also restrict access to delete or update that ResourceQuota; for example, by defining a ValidatingAdmissionPolicy.Users create resources (pods, services, etc.) in the namespace, and the quota system tracks usage to ensure it does not exceed hard resource limits defined in a ResourceQuota.You can apply a scope to a ResourceQuota to limit where it applies,If creating or updating a resource violates a quota constraint, the control plane rejects that request with HTTP status code 403 Forbidden. The error includes a message explaining the constraint that would have been violated.If quotas are enabled in a namespace for resource such as cpu and memory, users must specify requests or limits for those values when they define a Pod; otherwise, the quota system may reject pod creation.The resource quota walkthrough shows an example of how to avoid this problem.Note:You can define a LimitRange to force defaults on pods that make no compute resource requirements (so that users don't have to remember to do that).You often do not create Pods directly; for example, you more usually create a workload management object such as a Deployment. If you create a Deployment that tries to use more resources than are available, the creation of the Deployment (or other workload management object) succeeds, but the Deployment may not be able to get all of the Pods it manages to exist. In that case you can check the status of the Deployment, for example with kubectl describe, to see what has happened.For cpu and memory resources, ResourceQuotas enforce that every (new) pod in that namespace sets a limit for that resource. If you enforce a resource quota in a namespace for either cpu or memory, you and other clients, must specify either requests or limits for that resource, for every new Pod you submit. If you don't, the control plane may reject admission for that Pod.For other resources: ResourceQuota works and will ignore pods in the namespace without setting a limit or request for that resource. It means that you can create a new pod without limit/request for ephemeral storage if the resource quota limits the ephemeral storage of this namespace.You can use a LimitRange to automatically set a default request for these resources.The name of a ResourceQuota object must be a valid DNS subdomain name.Examples of policies that could be created using namespaces and quotas are:In a cluster with a capacity of 32 GiB RAM, and 16 cores, let team A use 20 GiB and 10 cores, let B use 10GiB and 4 cores, and hold 2GiB and 2 cores in reserve for future allocation.Limit the "testing" namespace to using 1 core and 1GiB RAM. Let the "production" namespace use any amount.In the case where the total capacity of the cluster is less than the sum of the quotas of the namespaces, there may be contention for resources. This is handled on a first-come-first-served basis.Enabling Resource QuotaResourceQuota support is enabled by default for many Kubernetes distributions. It is enabled when the API server --enable-admission-plugins= flag has ResourceQuota as one of its arguments.A resource quota is enforced in a particular namespace when there is a ResourceQuota in that namespace.Types of resource quotaThe ResourceQuota mechanism lets you enforce different kinds of limits. This section describes the types of limit that you can enforce.Quota for infrastructure resourcesYou can limit the total sum of compute resources that can be requested in a given namespace.The following resource types are supported:Resource NameDescriptionlimits.cpuAcross all pods in a non-terminal state, the sum of CPU limits cannot exceed this value.limits.memoryAcross all pods in a non-terminal state, the sum of memory limits cannot exceed this value.requests.cpuAcross all pods in a non-terminal state, the sum of CPU requests cannot exceed this value.requests.memoryAcross all pods in a non-terminal state, the sum of memory requests cannot exceed this value.hugepages-<size>Across all pods in a non-terminal state, the number of huge page requests of the specified size cannot exceed this value.cpuSame as requests.cpumemorySame as requests.memoryQuota for extended resourcesIn addition to the resources mentioned above, in release 1.10, quota support for extended resources is added.As overcommit is not allowed for extended resources, it makes no sense to specify both requests and limits for the same extended resource in a quota. So for extended resources, only quota items with prefix requests. are allowed.Take the GPU resource as an example, if the resource name is nvidia.com/gpu, and you want to limit the total number of GPUs requested in a namespace to 4, you can define a quota as follows:requests.nvidia.com/gpu: 4See Viewing and Setting Quotas for more details.Quota for storageYou can limit the total sum of storage for volumes that can be requested in a given namespace.In addition, you can limit consumption of storage resources based on associated StorageClass.Resource NameDescriptionrequests.storageAcross all persistent volume claims, the sum of storage requests cannot exceed this value.persistentvolumeclaimsThe total number of PersistentVolumeClaims that can exist in the namespace.<storage-class-name>.storageclass.storage.k8s.io/requests.storageAcross all persistent volume claims associated with the <storage-class-name>, the sum of storage requests cannot exceed this value.<storage-class-name>.storageclass.storage.k8s.io/persistentvolumeclaimsAcross all persistent volume claims associated with the <storage-class-name>, the total number of persistent volume claims that can exist in the namespace.For example, if you want to quota storage with gold StorageClass separate from a bronze StorageClass, you can define a quota as follows:gold.storageclass.storage.k8s.io/requests.storage: 500Gibronze.storageclass.storage.k8s.io/requests.storage: 100GiQuota for local ephemeral storageFEATURE STATE: Kubernetes v1.8 [alpha]Resource NameDescriptionrequests.ephemeral-storageAcross all pods in the namespace, the sum of local ephemeral storage requests cannot exceed this value.limits.ephemeral-storageAcross all pods in the namespace, the sum of local ephemeral storage limits cannot exceed this value.ephemeral-storageSame as requests.ephemeral-storage.Note:When using a CRI container runtime, container logs will count against the ephemeral storage quota. This can result in the unexpected eviction of pods that have exhausted their storage quotas.Refer to Logging Architecture for details.Quota on object countYou can set quota for the total number of one particular resource kind in the Kubernetes API, using the following syntax:count/<resource>.<group> for resources from non-core API groupscount/<resource> for resources from the core API groupFor example, the PodTemplate API is in the core API group and so if you want to limit the number of PodTemplate objects in a namespace, you use count/podtemplates.These types of quotas are useful to protect against exhaustion of control plane storage. For example, you may want to limit the number of Secrets in a server given their large size. Too many Secrets in a cluster can actually prevent servers and controllers from starting. You can set a quota for Jobs to protect against a poorly configured CronJob. CronJobs that create too many Jobs in a namespace can lead to a denial of service.If you define a quota this way, it applies to Kubernetes' APIs that are part of the API server, and to any custom resources backed by a CustomResourceDefinition. For example, to create a quota on a widgets custom resource in the example.com API group, use count/widgets.example.com. If you use API aggregation to add additional, custom APIs that are not defined as CustomResourceDefinitions, the core Kubernetes control plane does not enforce quota for the aggregated API. The extension API server is expected to provide quota enforcement if that's appropriate for the custom API.Generic syntaxThis is a list of common examples of object kinds that you may want to put under object count quota, listed by the configuration string that you would use.count/podscount/persistentvolumeclaimscount/servicescount/secretscount/configmapscount/deployments.appscount/replicasets.appscount/statefulsets.appscount/jobs.batchcount/cronjobs.batchSpecialized syntaxThere is another syntax only to set the same type of quota, that only works for certain API kinds. The following types are supported:Resource NameDescriptionconfigmapsThe total number of ConfigMaps that can exist in the namespace.persistentvolumeclaimsThe total number of PersistentVolumeClaims that can exist in the namespace.podsThe total number of Pods in a non-terminal state that can exist in the namespace. A pod is in a terminal state if .status.phase in (Failed, Succeeded) is true.replicationcontrollersThe total number of ReplicationControllers that can exist in the namespace.resourcequotasThe total number of ResourceQuotas that can exist in the namespace.servicesThe total number of Services that can exist in the namespace.services.loadbalancersThe total number of Services of type LoadBalancer that can exist in the namespace.services.nodeportsThe total number of NodePorts allocated to Services of type NodePort or LoadBalancer that can exist in the namespace.secretsThe total number of Secrets that can exist in the namespace.For example, pods quota counts and enforces a maximum on the number of pods created in a single namespace that are not terminal. You might want to set a pods quota on a namespace to avoid the case where a user creates many small pods and exhausts the cluster's supply of Pod IPs.You can find more examples on Viewing and Setting Quotas.Quota ScopesEach quota can have an associated set of scopes. A quota will only measure usage for a resource if it matches the intersection of enumerated scopes.When a scope is added to the quota, it limits the number of resources it supports to those that pertain to the scope. Resources specified on the quota outside of the allowed set results in a validation error.ScopeDescriptionTerminatingMatch pods where .spec.activeDeadlineSeconds >= 0NotTerminatingMatch pods where .spec.activeDeadlineSeconds is nilBestEffortMatch pods that have best effort quality of service.NotBestEffortMatch pods that do not have best effort quality of service.PriorityClassMatch pods that references the specified priority class.CrossNamespacePodAffinityMatch pods that have cross-namespace pod (anti)affinity terms.VolumeAttributesClassMatch persistentvolumeclaims that references the specified volume attributes class.The BestEffort scope restricts a quota to tracking the following resource:podsThe Terminating, NotTerminating, NotBestEffort and PriorityClass scopes restrict a quota to tracking the following resources:podscpumemoryrequests.cpurequests.memorylimits.cpulimits.memoryNote that you cannot specify both the Terminating and the NotTerminating scopes in the same quota, and you cannot specify both the BestEffort and NotBestEffort scopes in the same quota either.The scopeSelector supports the following values in the operator field:InNotInExistsDoesNotExistWhen using one of the following values as the scopeName when defining the scopeSelector, the operator must be Exists.TerminatingNotTerminatingBestEffortNotBestEffortIf the operator is In or NotIn, the values field must have at least one value. For example: scopeSelector: matchExpressions: - scopeName: PriorityClass operator: In values: - middle If the operator is Exists or DoesNotExist, the values field must NOT be specified.Resource Quota Per PriorityClassFEATURE STATE: Kubernetes v1.17 [stable]Pods can be created at a specific priority. You can control a pod's consumption of system resources based on a pod's priority, by using the scopeSelector field in the quota spec.A quota is matched and consumed only if scopeSelector in the quota spec selects the pod.When quota is scoped for priority class using scopeSelector field, quota object is restricted to track only following resources:podscpumemoryephemeral-storagelimits.cpulimits.memorylimits.ephemeral-storagerequests.cpurequests.memoryrequests.ephemeral-storageThis example creates a quota object and matches it with pods at specific priorities. The example works as follows:Pods in the cluster have one of the three priority classes, "low", "medium", "high".One quota object is created for each priority.Save the following YAML to a file quota.yaml.policy/quota.yaml apiVersion: v1 kind: List items: - apiVersion: v1 kind: ResourceQuota metadata: name: pods-high spec: hard: cpu: "1000" memory: "200Gi" pods: "10" scopeSelector: matchExpressions: - operator: In scopeName: PriorityClass values: ["high"] - apiVersion: v1 kind: ResourceQuota metadata: name: pods-medium spec: hard: cpu: "10" memory: "20Gi" pods: "10" scopeSelector: matchExpressions: - operator: In scopeName: PriorityClass values: ["medium"] - apiVersion: v1 kind: ResourceQuota metadata: name: pods-low spec: hard: cpu: "5" memory: "10Gi" pods: "10" scopeSelector: matchExpressions: - operator: In scopeName: PriorityClass values: ["low"] Apply the YAML using kubectl create.kubectl create -f ./quota.yaml resourcequota/pods-high created resourcequota/pods-medium created resourcequota/pods-low created Verify that Used quota is 0 using kubectl describe quota.kubectl describe quota Name: pods-high Namespace: default Resource Used Hard -------- ---- ---- cpu 0 1k memory 0 200Gi pods 0 10 Name: pods-low Namespace: default Resource Used Hard -------- ---- ---- cpu 0 5 memory 0 10Gi pods 0 10 Name: pods-medium Namespace: default Resource Used Hard -------- ---- ---- cpu 0 10 memory 0 20Gi pods 0 10 Create a pod with priority "high". Save the following YAML to a file high-priority-pod.yaml.policy/high-priority-pod.yaml apiVersion: v1 kind: Pod metadata: name: high-priority spec: containers: - name: high-priority image: ubuntu command: ["/bin/sh"] args: ["-c", "while true; do echo hello; sleep 10;done"] resources: requests: memory: "10Gi" cpu: "500m" limits: memory: "10Gi" cpu: "500m" priorityClassName: high Apply it with kubectl create.kubectl create -f ./high-priority-pod.yaml Verify that "Used" stats for "high" priority quota, pods-high, has changed and that the other two quotas are unchanged.kubectl describe quota Name: pods-high Namespace: default Resource Used Hard -------- ---- ---- cpu 500m 1k memory 10Gi 200Gi pods 1 10 Name: pods-low Namespace: default Resource Used Hard -------- ---- ---- cpu 0 5 memory 0 10Gi pods 0 10 Name: pods-medium Namespace: default Resource Used Hard -------- ---- ---- cpu 0 10 memory 0 20Gi pods 0 10 Cross-namespace Pod Affinity QuotaFEATURE STATE: Kubernetes v1.24 [stable]Operators can use CrossNamespacePodAffinity quota scope to limit which namespaces are allowed to have pods with affinity terms that cross namespaces. Specifically, it controls which pods are allowed to set namespaces or namespaceSelector fields in pod affinity terms.Preventing users from using cross-namespace affinity terms might be desired since a pod with anti-affinity constraints can block pods from all other namespaces from getting scheduled in a failure domain.Using this scope operators can prevent certain namespaces (foo-ns in the example below) from having pods that use cross-namespace pod affinity by creating a resource quota object in that namespace with CrossNamespacePodAffinity scope and hard limit of 0:apiVersion: v1 kind: ResourceQuota metadata: name: disable-cross-namespace-affinity namespace: foo-ns spec: hard: pods: "0" scopeSelector: matchExpressions: - scopeName: CrossNamespacePodAffinity operator: Exists If operators want to disallow using namespaces and namespaceSelector by default, and only allow it for specific namespaces, they could configure CrossNamespacePodAffinity as a limited resource by setting the kube-apiserver flag --admission-control-config-file to the path of the following configuration file:apiVersion: apiserver.config.k8s.io/v1 kind: AdmissionConfiguration plugins: - name: "ResourceQuota" configuration: apiVersion: apiserver.config.k8s.io/v1 kind: ResourceQuotaConfiguration limitedResources: - resource: pods matchScopes: - scopeName: CrossNamespacePodAffinity operator: Exists With the above configuration, pods can use namespaces and namespaceSelector in pod affinity only if the namespace where they are created have a resource quota object with CrossNamespacePodAffinity scope and a hard limit greater than or equal to the number of pods using those fields.Resource Quota Per VolumeAttributesClassFEATURE STATE: Kubernetes v1.34 [stable] (enabled by default: true)PersistentVolumeClaims can be created with a specific volume attributes class, and might be modified after creation. You can control a PVC's consumption of storage resources based on the associated volume attributes classes, by using the scopeSelector field in the quota spec.The PVC references the associated volume attributes class by the following fields:spec.volumeAttributesClassNamestatus.currentVolumeAttributesClassNamestatus.modifyVolumeStatus.targetVolumeAttributesClassNameA quota is matched and consumed only if scopeSelector in the quota spec selects the PVC.When the quota is scoped for the volume attributes class using the scopeSelector field, the quota object is restricted to track only the following resources:persistentvolumeclaimsrequests.storageThis example creates a quota object and matches it with PVC at specific volume attributes classes. The example works as follows:PVCs in the cluster have at least one of the three volume attributes classes, "gold", "silver", "copper".One quota object is created for each volume attributes class.Save the following YAML to a file quota-vac.yaml.policy/quota-vac.yaml apiVersion: v1 kind: List items: - apiVersion: v1 kind: ResourceQuota metadata: name: pvcs-gold spec: hard: requests.storage: "10Gi" persistentvolumeclaims: "10" scopeSelector: matchExpressions: - operator: In scopeName: VolumeAttributesClass values: ["gold"] - apiVersion: v1 kind: ResourceQuota metadata: name: pvcs-silver spec: hard: requests.storage: "20Gi" persistentvolumeclaims: "10" scopeSelector: matchExpressions: - operator: In scopeName: VolumeAttributesClass values: ["silver"] - apiVersion: v1 kind: ResourceQuota metadata: name: pvcs-copper spec: hard: requests.storage: "30Gi" persistentvolumeclaims: "10" scopeSelector: matchExpressions: - operator: In scopeName: VolumeAttributesClass values: ["copper"] Apply the YAML using kubectl create.kubectl create -f ./quota-vac.yaml resourcequota/pvcs-gold created resourcequota/pvcs-silver created resourcequota/pvcs-copper created Verify that Used quota is 0 using kubectl describe quota.kubectl describe quota Name: pvcs-gold Namespace: default Resource Used Hard -------- ---- ---- persistentvolumeclaims 0 10 requests.storage 0 10Gi Name: pvcs-silver Namespace: default Resource Used Hard -------- ---- ---- persistentvolumeclaims 0 10 requests.storage 0 20Gi Name: pvcs-copper Namespace: default Resource Used Hard -------- ---- ---- persistentvolumeclaims 0 10 requests.storage 0 30Gi Create a pvc with volume attributes class "gold". Save the following YAML to a file gold-vac-pvc.yaml.policy/gold-vac-pvc.yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: gold-vac-pvc spec: accessModes: - ReadWriteOnce resources: requests: storage: 2Gi storageClassName: # change this to the name of the storage class you want to use volumeAttributesClassName: gold Apply it with kubectl create.kubectl create -f ./gold-vac-pvc.yaml Verify that "Used" stats for "gold" volume attributes class quota, pvcs-gold has changed and that the other two quotas are unchanged.kubectl describe quota Name: pvcs-gold Namespace: default Resource Used Hard -------- ---- ---- persistentvolumeclaims 1 10 requests.storage 2Gi 10Gi Name: pvcs-silver Namespace: default Resource Used Hard -------- ---- ---- persistentvolumeclaims 0 10 requests.storage 0 20Gi Name: pvcs-copper Namespace: default Resource Used Hard -------- ---- ---- persistentvolumeclaims 0 10 requests.storage 0 30Gi Once the PVC is bound, it is allowed to modify the desired volume attributes class. Let's change it to "silver" with kubectl patch.kubectl patch pvc gold-vac-pvc --type='merge' -p '{"spec":{"volumeAttributesClassName":"silver"}}' Verify that "Used" stats for "silver" volume attributes class quota, pvcs-silver has changed, pvcs-copper is unchanged, and pvcs-gold might be unchanged or released, which depends on the PVC's status.kubectl describe quota Name: pvcs-gold Namespace: default Resource Used Hard -------- ---- ---- persistentvolumeclaims 1 10 requests.storage 2Gi 10Gi Name: pvcs-silver Namespace: default Resource Used Hard -------- ---- ---- persistentvolumeclaims 1 10 requests.storage 2Gi 20Gi Name: pvcs-copper Namespace: default Resource Used Hard -------- ---- ---- persistentvolumeclaims 0 10 requests.storage 0 30Gi Let's change it to "copper" with kubectl patch.kubectl patch pvc gold-vac-pvc --type='merge' -p '{"spec":{"volumeAttributesClassName":"copper"}}' Verify that "Used" stats for "copper" volume attributes class quota, pvcs-copper has changed, pvcs-silver and pvcs-gold might be unchanged or released, which depends on the PVC's status.kubectl describe quota Name: pvcs-gold Namespace: default Resource Used Hard -------- ---- ---- persistentvolumeclaims 1 10 requests.storage 2Gi 10Gi Name: pvcs-silver Namespace: default Resource Used Hard -------- ---- ---- persistentvolumeclaims 1 10 requests.storage 2Gi 20Gi Name: pvcs-copper Namespace: default Resource Used Hard -------- ---- ---- persistentvolumeclaims 1 10 requests.storage 2Gi 30Gi Print the manifest of the PVC using the following command:kubectl get pvc gold-vac-pvc -o yaml It might show the following output:apiVersion: v1 kind: PersistentVolumeClaim metadata: name: gold-vac-pvc spec: accessModes: - ReadWriteOnce resources: requests: storage: 2Gi storageClassName: default volumeAttributesClassName: copper status: accessModes: - ReadWriteOnce capacity: storage: 2Gi currentVolumeAttributesClassName: gold phase: Bound modifyVolumeStatus: status: InProgress targetVolumeAttributesClassName: silver storageClassName: default Wait a moment for the volume modification to complete, then verify the quota again.kubectl describe quota Name: pvcs-gold Namespace: default Resource Used Hard -------- ---- ---- persistentvolumeclaims 0 10 requests.storage 0 10Gi Name: pvcs-silver Namespace: default Resource Used Hard -------- ---- ---- persistentvolumeclaims 0 10 requests.storage 0 20Gi Name: pvcs-copper Namespace: default Resource Used Hard -------- ---- ---- persistentvolumeclaims 1 10 requests.storage 2Gi 30Gi Requests compared to LimitsWhen allocating compute resources, each container may specify a request and a limit value for either CPU or memory. The quota can be configured to quota either value.If the quota has a value specified for requests.cpu or requests.memory, then it requires that every incoming container makes an explicit request for those resources. If the quota has a value specified for limits.cpu or limits.memory, then it requires that every incoming container specifies an explicit limit for those resources.Viewing and Setting Quotaskubectl supports creating, updating, and viewing quotas:kubectl create namespace myspace cat <<EOF > compute-resources.yaml apiVersion: v1 kind: ResourceQuota metadata: name: compute-resources spec: hard: requests.cpu: "1" requests.memory: "1Gi" limits.cpu: "2" limits.memory: "2Gi" requests.nvidia.com/gpu: 4 EOF kubectl create -f ./compute-resources.yaml --namespace=myspace cat <<EOF > object-counts.yaml apiVersion: v1 kind: ResourceQuota metadata: name: object-counts spec: hard: configmaps: "10" persistentvolumeclaims: "4" pods: "4" replicationcontrollers: "20" secrets: "10" services: "10" services.loadbalancers: "2" EOF kubectl create -f ./object-counts.yaml --namespace=myspace kubectl get quota --namespace=myspace NAME AGE compute-resources 30s object-counts 32s kubectl describe quota compute-resources --namespace=myspace Name: compute-resources Namespace: myspace Resource Used Hard -------- ---- ---- limits.cpu 0 2 limits.memory 0 2Gi requests.cpu 0 1 requests.memory 0 1Gi requests.nvidia.com/gpu 0 4 kubectl describe quota object-counts --namespace=myspace Name: object-counts Namespace: myspace Resource Used Hard -------- ---- ---- configmaps 0 10 persistentvolumeclaims 0 4 pods 0 4 replicationcontrollers 0 20 secrets 1 10 services 0 10 services.loadbalancers 0 2 kubectl also supports object count quota for all standard namespaced resources using the syntax count/<resource>.<group>:kubectl create namespace myspace kubectl create quota test --hard=count/deployments.apps=2,count/replicasets.apps=4,count/pods=3,count/secrets=4 --namespace=myspace kubectl create deployment nginx --image=nginx --namespace=myspace --replicas=2 kubectl describe quota --namespace=myspace Name: test Namespace: myspace Resource Used Hard -------- ---- ---- count/deployments.apps 1 2 count/pods 2 3 count/replicasets.apps 1 4 count/secrets 1 4 Quota and Cluster CapacityResourceQuotas are independent of the cluster capacity. They are expressed in absolute units. So, if you add nodes to your cluster, this does not automatically give each namespace the ability to consume more resources.Sometimes more complex policies may be desired, such as:Proportionally divide total cluster resources among several teams.Allow each tenant to grow resource usage as needed, but have a generous limit to prevent accidental resource exhaustion.Detect demand from one namespace, add nodes, and increase quota.Such policies could be implemented using ResourceQuotas as building blocks, by writing a "controller" that watches the quota usage and adjusts the quota hard limits of each namespace according to other signals.Note that resource quota divides up aggregate cluster resources, but it creates no restrictions around nodes: pods from several namespaces may run on the same node.Limit Priority Class consumption by defaultIt may be desired that pods at a particular priority, such as "cluster-services", should be allowed in a namespace, if and only if, a matching quota object exists.With this mechanism, operators are able to restrict usage of certain high priority classes to a limited number of namespaces and not every namespace will be able to consume these priority classes by default.To enforce this, kube-apiserver flag --admission-control-config-file should be used to pass path to the following configuration file:apiVersion: apiserver.config.k8s.io/v1 kind: AdmissionConfiguration plugins: - name: "ResourceQuota" configuration: apiVersion: apiserver.config.k8s.io/v1 kind: ResourceQuotaConfiguration limitedResources: - resource: pods matchScopes: - scopeName: PriorityClass operator: In values: ["cluster-services"] Then, create a resource quota object in the kube-system namespace:policy/priority-class-resourcequota.yaml apiVersion: v1 kind: ResourceQuota metadata: name: pods-cluster-services spec: scopeSelector: matchExpressions: - operator : In scopeName: PriorityClass values: ["cluster-services"]kubectl apply -f https://k8s.io/examples/policy/priority-class-resourcequota.yaml -n kube-system resourcequota/pods-cluster-services created In this case, a pod creation will be allowed if:the Pod's priorityClassName is not specified.the Pod's priorityClassName is specified to a value other than cluster-services.the Pod's priorityClassName is set to cluster-services, it is to be created in the kube-system namespace, and it has passed the resource quota check.A Pod creation request is rejected if its priorityClassName is set to cluster-services and it is to be created in a namespace other than kube-system.What's nextSee a detailed example for how to use resource quota.Read the ResourceQuota API referenceLearn about LimitRangesYou can read the historical ResourceQuota design document for more information.You can also read the Quota support for priority class design document.

Resource Quotas


Configuring swap memory on Kubernetes nodesThis page provides an example of how to provision and configure swap memory on a Kubernetes node using kubeadm.ObjectivesProvision swap memory on a Kubernetes node using kubeadm.Learn to configure both encrypted and unencrypted swap.Learn to enable swap on boot.Before you beginYou need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. It is recommended to run this tutorial on a cluster with at least two nodes that are not acting as control plane hosts. If you do not already have a cluster, you can create one by using minikube or you can use one of these Kubernetes playgrounds:iximiuz LabsKillercodaKodeKloudPlay with KubernetesYour Kubernetes server must be at or later than version 1.33.To check the version, enter kubectl version.You need at least one worker node in your cluster which needs to run a Linux operating system. It is required for this demo that the kubeadm tool be installed, following the steps outlined in the kubeadm installation guide.On each worker node where you will configure swap use, you need:fallocatemkswapswaponFor encrypted swap space (recommended), you also need:cryptsetupInstall a swap-enabled cluster with kubeadmCreate a swap file and turn swap onIf swap is not enabled, there's a need to provision swap on the node. The following sections demonstrate creating 4GiB of swap, both in the encrypted and unencrypted case.An encrypted swap file can be set up as follows. Bear in mind that this example uses the cryptsetup binary (which is available on most Linux distributions).# Allocate storage and restrict access fallocate --length 4GiB /swapfile chmod 600 /swapfile # Create an encrypted device backed by the allocated storage cryptsetup --type plain --cipher aes-xts-plain64 --key-size 256 -d /dev/urandom open /swapfile cryptswap # Format the swap space mkswap /dev/mapper/cryptswap # Activate the swap space for paging swapon /dev/mapper/cryptswap An unencrypted swap file can be set up as follows.# Allocate storage and restrict access fallocate --length 4GiB /swapfile chmod 600 /swapfile # Format the swap space mkswap /swapfile # Activate the swap space for paging swapon /swapfile Verify that swap is enabledSwap can be verified to be enabled with both swapon -s command or the free command.Using swapon -s:Filename Type Size Used Priority /dev/dm-0 partition 4194300 0 -2 Using free -h: total used free shared buff/cache available Mem: 3.8Gi 1.3Gi 249Mi 25Mi 2.5Gi 2.5Gi Swap: 4.0Gi 0B 4.0Gi Enable swap on bootAfter setting up swap, to start the swap file at boot time, you typically either set up a systemd unit to activate (encrypted) swap, or you add a line similar to /swapfile swap swap defaults 0 0 into /etc/fstab.Using systemd for swap activation allows the system to delay kubelet start until swap is available, if that is something you want to ensure. In a similar way, using systemd allows your server to leave swap active until kubelet (and, typically, your container runtime) have shut down.Set up kubelet configurationAfter enabling swap on the node, kubelet needs to be configured to use it. You need to select a swap behavior for this node. You'll configure LimitedSwap behavior for this tutorial.Find and edit the kubelet configuration file, and:set failSwapOn to falseset memorySwap.swapBehavior to LimitedSwap # this fragment goes into the kubelet's configuration file failSwapOn: false memorySwap: swapBehavior: LimitedSwap In order for these configurations to take effect, kubelet needs to be restarted. Typically you do that by running:systemctl restart kubelet.service You should find that the kubelet is now healthy, and that you can run Pods that use swap memory as needed.


Changing The Kubernetes Package RepositoryThis page explains how to enable a package repository for the desired Kubernetes minor release upon upgrading a cluster. This is only needed for users of the community-owned package repositories hosted at pkgs.k8s.io. Unlike the legacy package repositories, the community-owned package repositories are structured in a way that there's a dedicated package repository for each Kubernetes minor version.Note:This guide only covers a part of the Kubernetes upgrade process. Please see the upgrade guide for more information about upgrading Kubernetes clusters.Note:This step is only needed upon upgrading a cluster to another minor release. If you're upgrading to another patch release within the same minor release (e.g. v1.34.5 to v1.34.7), you don't need to follow this guide. However, if you're still using the legacy package repositories, you'll need to migrate to the new community-owned package repositories before upgrading (see the next section for more details on how to do this).Before you beginThis document assumes that you're already using the community-owned package repositories (pkgs.k8s.io). If that's not the case, it's strongly recommended to migrate to the community-owned package repositories as described in the official announcement.Note: The legacy package repositories (apt.kubernetes.io and yum.kubernetes.io) have been deprecated and frozen starting from September 13, 2023. Using the new package repositories hosted at pkgs.k8s.io is strongly recommended and required in order to install Kubernetes versions released after September 13, 2023. The deprecated legacy repositories, and their contents, might be removed at any time in the future and without a further notice period. The new package repositories provide downloads for Kubernetes versions starting with v1.24.0.Verifying if the Kubernetes package repositories are usedIf you're unsure whether you're using the community-owned package repositories or the legacy package repositories, take the following steps to verify:Print the contents of the file that defines the Kubernetes apt repository:# On your system, this configuration file could have a different name pager /etc/apt/sources.list.d/kubernetes.list If you see a line similar to:deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.33/deb/ / You're using the Kubernetes package repositories and this guide applies to you. Otherwise, it's strongly recommended to migrate to the Kubernetes package repositories as described in the official announcement.Print the contents of the file that defines the Kubernetes yum repository:# On your system, this configuration file could have a different name cat /etc/yum.repos.d/kubernetes.repo If you see a baseurl similar to the baseurl in the output below:[kubernetes] name=Kubernetes baseurl=https://pkgs.k8s.io/core:/stable:/v1.33/rpm/ enabled=1 gpgcheck=1 gpgkey=https://pkgs.k8s.io/core:/stable:/v1.33/rpm/repodata/repomd.xml.key exclude=kubelet kubeadm kubectl You're using the Kubernetes package repositories and this guide applies to you. Otherwise, it's strongly recommended to migrate to the Kubernetes package repositories as described in the official announcement.Print the contents of the file that defines the Kubernetes zypper repository:# On your system, this configuration file could have a different name cat /etc/zypp/repos.d/kubernetes.repo If you see a baseurl similar to the baseurl in the output below:[kubernetes] name=Kubernetes baseurl=https://pkgs.k8s.io/core:/stable:/v1.33/rpm/ enabled=1 gpgcheck=1 gpgkey=https://pkgs.k8s.io/core:/stable:/v1.33/rpm/repodata/repomd.xml.key exclude=kubelet kubeadm kubectl You're using the Kubernetes package repositories and this guide applies to you. Otherwise, it's strongly recommended to migrate to the Kubernetes package repositories as described in the official announcement.Note:The URL used for the Kubernetes package repositories is not limited to pkgs.k8s.io, it can also be one of:pkgs.k8s.iopkgs.kubernetes.iopackages.kubernetes.ioSwitching to another Kubernetes package repositoryThis step should be done upon upgrading from one to another Kubernetes minor release in order to get access to the packages of the desired Kubernetes minor version.Open the file that defines the Kubernetes apt repository using a text editor of your choice:nano /etc/apt/sources.list.d/kubernetes.list You should see a single line with the URL that contains your current Kubernetes minor version. For example, if you're using v1.33, you should see this:deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.33/deb/ / Change the version in the URL to the next available minor release, for example:deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.34/deb/ / Save the file and exit your text editor. Continue following the relevant upgrade instructions.Open the file that defines the Kubernetes yum repository using a text editor of your choice:nano /etc/yum.repos.d/kubernetes.repo You should see a file with two URLs that contain your current Kubernetes minor version. For example, if you're using v1.33, you should see this:[kubernetes] name=Kubernetes baseurl=https://pkgs.k8s.io/core:/stable:/v1.33/rpm/ enabled=1 gpgcheck=1 gpgkey=https://pkgs.k8s.io/core:/stable:/v1.33/rpm/repodata/repomd.xml.key exclude=kubelet kubeadm kubectl cri-tools kubernetes-cni Change the version in these URLs to the next available minor release, for example:[kubernetes] name=Kubernetes baseurl=https://pkgs.k8s.io/core:/stable:/v1.34/rpm/ enabled=1 gpgcheck=1 gpgkey=https://pkgs.k8s.io/core:/stable:/v1.34/rpm/repodata/repomd.xml.key exclude=kubelet kubeadm kubectl cri-tools kubernetes-cni Save the file and exit your text editor. Continue following the relevant upgrade instructions.What's nextSee how to Upgrade Linux nodes.See how to Upgrade Windows nodes.


Resource metrics pipelineFor Kubernetes, the Metrics API offers a basic set of metrics to support automatic scaling and similar use cases. This API makes information available about resource usage for node and pod, including metrics for CPU and memory. If you deploy the Metrics API into your cluster, clients of the Kubernetes API can then query for this information, and you can use Kubernetes' access control mechanisms to manage permissions to do so.The HorizontalPodAutoscaler (HPA) and VerticalPodAutoscaler (VPA) use data from the metrics API to adjust workload replicas and resources to meet customer demand.You can also view the resource metrics using the kubectl top command.Note:The Metrics API, and the metrics pipeline that it enables, only offers the minimum CPU and memory metrics to enable automatic scaling using HPA and / or VPA. If you would like to provide a more complete set of metrics, you can complement the simpler Metrics API by deploying a second metrics pipeline that uses the Custom Metrics API.Figure 1 illustrates the architecture of the resource metrics pipeline.flowchart RL subgraph cluster[Cluster] direction RL S[] A[Metrics-Server] subgraph B[Nodes] direction TB D[cAdvisor] --> C[kubelet] E[Containerruntime] --> D E1[Containerruntime] --> D P[pod data] -.- C end L[APIserver] W[HPA] C ---->|node levelresource metrics| A -->|metricsAPI| L --> W end L ---> K[kubectltop] classDef box fill:#fff,stroke:#000,stroke-width:1px,color:#000; class W,B,P,K,cluster,D,E,E1 box classDef spacewhite fill:#ffffff,stroke:#fff,stroke-width:0px,color:#000 class S spacewhite classDef k8s fill:#326ce5,stroke:#fff,stroke-width:1px,color:#fff; class A,L,C k8sJavaScript must be enabled to view this contentFigure 1. Resource Metrics PipelineThe architecture components, from right to left in the figure, consist of the following:cAdvisor: Daemon for collecting, aggregating and exposing container metrics included in Kubelet.kubelet: Node agent for managing container resources. Resource metrics are accessible using the /metrics/resource and /stats kubelet API endpoints.node level resource metrics: API provided by the kubelet for discovering and retrieving per-node summarized stats available through the /metrics/resource endpoint.metrics-server: Cluster addon component that collects and aggregates resource metrics pulled from each kubelet. The API server serves Metrics API for use by HPA, VPA, and by the kubectl top command. Metrics Server is a reference implementation of the Metrics API.Metrics API: Kubernetes API supporting access to CPU and memory used for workload autoscaling. To make this work in your cluster, you need an API extension server that provides the Metrics API.Note:cAdvisor supports reading metrics from cgroups, which works with typical container runtimes on Linux. If you use a container runtime that uses another resource isolation mechanism, for example virtualization, then that container runtime must support CRI Container Metrics in order for metrics to be available to the kubelet.Metrics APIFEATURE STATE: Kubernetes 1.8 [beta]The metrics-server implements the Metrics API. This API allows you to access CPU and memory usage for the nodes and pods in your cluster. Its primary role is to feed resource usage metrics to K8s autoscaler components.Here is an example of the Metrics API request for a minikube node piped through jq for easier reading:kubectl get --raw "/apis/metrics.k8s.io/v1beta1/nodes/minikube" | jq '.' Here is the same API call using curl:curl http://localhost:8080/apis/metrics.k8s.io/v1beta1/nodes/minikube Sample response:{ "kind": "NodeMetrics", "apiVersion": "metrics.k8s.io/v1beta1", "metadata": { "name": "minikube", "selfLink": "/apis/metrics.k8s.io/v1beta1/nodes/minikube", "creationTimestamp": "2022-01-27T18:48:43Z" }, "timestamp": "2022-01-27T18:48:33Z", "window": "30s", "usage": { "cpu": "487558164n", "memory": "732212Ki" } } Here is an example of the Metrics API request for a kube-scheduler-minikube pod contained in the kube-system namespace and piped through jq for easier reading:kubectl get --raw "/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/kube-scheduler-minikube" | jq '.' Here is the same API call using curl:curl http://localhost:8080/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/kube-scheduler-minikube Sample response:{ "kind": "PodMetrics", "apiVersion": "metrics.k8s.io/v1beta1", "metadata": { "name": "kube-scheduler-minikube", "namespace": "kube-system", "selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/kube-scheduler-minikube", "creationTimestamp": "2022-01-27T19:25:00Z" }, "timestamp": "2022-01-27T19:24:31Z", "window": "30s", "containers": [ { "name": "kube-scheduler", "usage": { "cpu": "9559630n", "memory": "22244Ki" } } ] } The Metrics API is defined in the k8s.io/metrics repository. You must enable the API aggregation layer and register an APIService for the metrics.k8s.io API.To learn more about the Metrics API, see resource metrics API design, the metrics-server repository and the resource metrics API.Note:You must deploy the metrics-server or alternative adapter that serves the Metrics API to be able to access it.Measuring resource usageCPUCPU is reported as the average core usage measured in cpu units. One cpu, in Kubernetes, is equivalent to 1 vCPU/Core for cloud providers, and 1 hyper-thread on bare-metal Intel processors.This value is derived by taking a rate over a cumulative CPU counter provided by the kernel (in both Linux and Windows kernels). The time window used to calculate CPU is shown under window field in Metrics API.To learn more about how Kubernetes allocates and measures CPU resources, see meaning of CPU.MemoryMemory is reported as the working set, measured in bytes, at the instant the metric was collected.In an ideal world, the "working set" is the amount of memory in-use that cannot be freed under memory pressure. However, calculation of the working set varies by host OS, and generally makes heavy use of heuristics to produce an estimate.The Kubernetes model for a container's working set expects that the container runtime counts anonymous memory associated with the container in question. The working set metric typically also includes some cached (file-backed) memory, because the host OS cannot always reclaim pages.To learn more about how Kubernetes allocates and measures memory resources, see meaning of memory.Metrics ServerThe metrics-server fetches resource metrics from the kubelets and exposes them in the Kubernetes API server through the Metrics API for use by the HPA and VPA. You can also view these metrics using the kubectl top command.The metrics-server uses the Kubernetes API to track nodes and pods in your cluster. The metrics-server queries each node over HTTP to fetch metrics. The metrics-server also builds an internal view of pod metadata, and keeps a cache of pod health. That cached pod health information is available via the extension API that the metrics-server makes available.For example with an HPA query, the metrics-server needs to identify which pods fulfill the label selectors in the deployment.The metrics-server calls the kubelet API to collect metrics from each node. Depending on the metrics-server version it uses:Metrics resource endpoint /metrics/resource in version v0.6.0+ orSummary API endpoint /stats/summary in older versionsWhat's nextTo learn more about the metrics-server, see the metrics-server repository.You can also check out the following:metrics-server designmetrics-server FAQmetrics-server known issuesmetrics-server releasesHorizontal Pod AutoscalingTo learn about how the kubelet serves node metrics, and how you can access those via the Kubernetes API, read Node Metrics Data.

Resource metrics pipeline


kubectl create service loadbalancerSynopsisCreate a LoadBalancer service with the specified name.kubectl create service loadbalancer NAME [--tcp=port:targetPort] [--dry-run=server|client|none] Examples # Create a new LoadBalancer service named my-lbs kubectl create service loadbalancer my-lbs --tcp=5678:8080 Options--allow-missing-template-keys Default: trueIf true, ignore any errors in templates when a field or map key is missing in the template. Only applies to golang and jsonpath output formats.--dry-run string[="unchanged"] Default: "none"Must be "none", "server", or "client". If client strategy, only print the object that would be sent, without sending it. If server strategy, submit server-side request without persisting the resource.--field-manager string Default: "kubectl-create"Name of the manager used to track field ownership.-h, --helphelp for loadbalancer-o, --output stringOutput format. One of: (json, yaml, name, go-template, go-template-file, template, templatefile, jsonpath, jsonpath-as-json, jsonpath-file).--save-configIf true, the configuration of current object will be saved in its annotation. Otherwise, the annotation will be unchanged. This flag is useful when you want to perform kubectl apply on this object in the future.--show-managed-fieldsIf true, keep the managedFields when printing objects in JSON or YAML format.--tcp stringsPort pairs can be specified as '<port>:<targetPort>'.--template stringTemplate string or path to template file to use when -o=go-template, -o=go-template-file. The template format is golang templates [http://golang.org/pkg/text/template/#pkg-overview].--validate string[="strict"] Default: "strict"Must be one of: strict (or true), warn, ignore (or false). "true" or "strict" will use a schema to validate the input and fail the request if invalid. It will perform server side validation if ServerSideFieldValidation is enabled on the api-server, but will fall back to less reliable client-side validation if not. "warn" will warn about unknown or duplicate fields without blocking the request if server-side field validation is enabled on the API server, and behave as "ignore" otherwise. "false" or "ignore" will not perform any schema validation, silently dropping any unknown or duplicate fields.Parent Options Inherited--as stringUsername to impersonate for the operation. User could be a regular user or a service account in a namespace.--as-group stringsGroup to impersonate for the operation, this flag can be repeated to specify multiple groups.--as-uid stringUID to impersonate for the operation.--cache-dir string Default: "$HOME/.kube/cache"Default cache directory--certificate-authority stringPath to a cert file for the certificate authority--client-certificate stringPath to a client certificate file for TLS--client-key stringPath to a client key file for TLS--cluster stringThe name of the kubeconfig cluster to use--context stringThe name of the kubeconfig context to use--disable-compressionIf true, opt-out of response compression for all requests to the server--insecure-skip-tls-verifyIf true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure--kubeconfig stringPath to the kubeconfig file to use for CLI requests.--kuberc stringPath to the kuberc file to use for preferences. This can be disabled by exporting KUBECTL_KUBERC=false feature gate or turning off the feature KUBERC=off.--match-server-versionRequire server version to match client version-n, --namespace stringIf present, the namespace scope for this CLI request--password stringPassword for basic authentication to the API server--profile string Default: "none"Name of profile to capture. One of (none|cpu|heap|goroutine|threadcreate|block|mutex)--profile-output string Default: "profile.pprof"Name of the file to write the profile to--request-timeout string Default: "0"The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests.-s, --server stringThe address and port of the Kubernetes API server--storage-driver-buffer-duration duration Default: 1m0sWrites in the storage driver will be buffered for this duration, and committed to the non memory backends as a single transaction--storage-driver-db string Default: "cadvisor"database name--storage-driver-host string Default: "localhost:8086"database host:port--storage-driver-password string Default: "root"database password--storage-driver-secureuse secure connection with database--storage-driver-table string Default: "stats"table name--storage-driver-user string Default: "root"database username--tls-server-name stringServer name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used--token stringBearer token for authentication to the API server--user stringThe name of the kubeconfig user to use--username stringUsername for basic authentication to the API server--version version[=true]--version, --version=raw prints version information and quits; --version=vX.Y.Z... sets the reported version--warnings-as-errorsTreat warnings received from the server as errors and exit with a non-zero exit codeSee Alsokubectl create service - Create a service using a specified subcommandThis page is automatically generated.If you plan to report an issue with this page, mention that the page is auto-generated in your issue description. The fix may need to happen elsewhere in the Kubernetes project.

kubectl create service loadbalancer


Configure Pod InitializationThis page shows how to use an Init Container to initialize a Pod before an application Container runs.Before you beginYou need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. It is recommended to run this tutorial on a cluster with at least two nodes that are not acting as control plane hosts. If you do not already have a cluster, you can create one by using minikube or you can use one of these Kubernetes playgrounds:iximiuz LabsKillercodaKodeKloudPlay with KubernetesTo check the version, enter kubectl version.Create a Pod that has an Init ContainerIn this exercise you create a Pod that has one application Container and one Init Container. The init container runs to completion before the application container starts.Here is the configuration file for the Pod:pods/init-containers.yaml apiVersion: v1 kind: Pod metadata: name: init-demo spec: containers: - name: nginx image: nginx ports: - containerPort: 80 volumeMounts: - name: workdir mountPath: /usr/share/nginx/html # These containers are run during pod initialization initContainers: - name: install image: busybox:1.28 command: - wget - "-O" - "/work-dir/index.html" - http://info.cern.ch volumeMounts: - name: workdir mountPath: "/work-dir" dnsPolicy: Default volumes: - name: workdir emptyDir: {} In the configuration file, you can see that the Pod has a Volume that the init container and the application container share.The init container mounts the shared Volume at /work-dir, and the application container mounts the shared Volume at /usr/share/nginx/html. The init container runs the following command and then terminates:wget -O /work-dir/index.html http://info.cern.ch Notice that the init container writes the index.html file in the root directory of the nginx server.Create the Pod:kubectl apply -f https://k8s.io/examples/pods/init-containers.yaml Verify that the nginx container is running:kubectl get pod init-demo The output shows that the nginx container is running:NAME READY STATUS RESTARTS AGE init-demo 1/1 Running 0 1m Get a shell into the nginx container running in the init-demo Pod:kubectl exec -it init-demo -- /bin/bash In your shell, send a GET request to the nginx server:root@nginx:~# apt-get update root@nginx:~# apt-get install curl root@nginx:~# curl localhost The output shows that nginx is serving the web page that was written by the init container: <title>http://info.cern.ch</title>
http://info.cern.ch - home of the first website
 ...
Browse the first website
 ... What's nextLearn more about communicating between Containers running in the same Pod.Learn more about Init Containers.Learn more about Volumes.Learn more about Debugging Init Containers

Configure Pod Initialization


Enforce Pod Security Standards by Configuring the Built-in Admission ControllerKubernetes provides a built-in admission controller to enforce the Pod Security Standards. You can configure this admission controller to set cluster-wide defaults and exemptions.Before you beginFollowing an alpha release in Kubernetes v1.22, Pod Security Admission became available by default in Kubernetes v1.23, as a beta. From version 1.25 onwards, Pod Security Admission is generally available.To check the version, enter kubectl version.If you are not running Kubernetes 1.34, you can switch to viewing this page in the documentation for the Kubernetes version that you are running.Configure the Admission ControllerNote:pod-security.admission.config.k8s.io/v1 configuration requires v1.25+. For v1.23 and v1.24, use v1beta1. For v1.22, use v1alpha1.apiVersion: apiserver.config.k8s.io/v1 kind: AdmissionConfiguration plugins: - name: PodSecurity configuration: apiVersion: pod-security.admission.config.k8s.io/v1 # see compatibility note kind: PodSecurityConfiguration # Defaults applied when a mode label is not set. # # Level label values must be one of: # - "privileged" (default) # - "baseline" # - "restricted" # # Version label values must be one of: # - "latest" (default) # - specific version like "v1.34" defaults: enforce: "privileged" enforce-version: "latest" audit: "privileged" audit-version: "latest" warn: "privileged" warn-version: "latest" exemptions: # Array of authenticated usernames to exempt. usernames: [] # Array of runtime class names to exempt. runtimeClasses: [] # Array of namespaces to exempt. namespaces: [] Note:The above manifest needs to be specified via the --admission-control-config-file to kube-apiserver.


ResourceClaimTemplateResourceClaimTemplate is used to produce ResourceClaim objects.apiVersion: resource.k8s.io/v1import "k8s.io/api/resource/v1"ResourceClaimTemplateResourceClaimTemplate is used to produce ResourceClaim objects.This is an alpha type and requires enabling the DynamicResourceAllocation feature gate.apiVersion: resource.k8s.io/v1kind: ResourceClaimTemplatemetadata (ObjectMeta)Standard object metadataspec (ResourceClaimTemplateSpec), requiredDescribes the ResourceClaim that is to be generated.This field is immutable. A ResourceClaim will get created by the control plane for a Pod when needed and then not get updated anymore.ResourceClaimTemplateSpecResourceClaimTemplateSpec contains the metadata and fields for a ResourceClaim.spec (ResourceClaimSpec), requiredSpec for the ResourceClaim. The entire content is copied unchanged into the ResourceClaim that gets created from this template. The same fields as in a ResourceClaim are also valid here.metadata (ObjectMeta)ObjectMeta may contain labels and annotations that will be copied into the ResourceClaim when creating it. No other fields are allowed and will be rejected during validation.ResourceClaimTemplateListResourceClaimTemplateList is a collection of claim templates.apiVersion: resource.k8s.io/v1kind: ResourceClaimTemplateListmetadata (ListMeta)Standard list metadataitems ([]ResourceClaimTemplate), requiredItems is the list of resource claim templates.Operationsget read the specified ResourceClaimTemplateHTTP RequestGET /apis/resource.k8s.io/v1/namespaces/{namespace}/resourceclaimtemplates/{name}Parametersname (in path): string, requiredname of the ResourceClaimTemplatenamespace (in path): string, requirednamespacepretty (in query): stringprettyResponse200 (ResourceClaimTemplate): OK401: Unauthorizedlist list or watch objects of kind ResourceClaimTemplateHTTP RequestGET /apis/resource.k8s.io/v1/namespaces/{namespace}/resourceclaimtemplatesParametersnamespace (in path): string, requirednamespaceallowWatchBookmarks (in query): booleanallowWatchBookmarkscontinue (in query): stringcontinuefieldSelector (in query): stringfieldSelectorlabelSelector (in query): stringlabelSelectorlimit (in query): integerlimitpretty (in query): stringprettyresourceVersion (in query): stringresourceVersionresourceVersionMatch (in query): stringresourceVersionMatchsendInitialEvents (in query): booleansendInitialEventstimeoutSeconds (in query): integertimeoutSecondswatch (in query): booleanwatchResponse200 (ResourceClaimTemplateList): OK401: Unauthorizedlist list or watch objects of kind ResourceClaimTemplateHTTP RequestGET /apis/resource.k8s.io/v1/resourceclaimtemplatesParametersallowWatchBookmarks (in query): booleanallowWatchBookmarkscontinue (in query): stringcontinuefieldSelector (in query): stringfieldSelectorlabelSelector (in query): stringlabelSelectorlimit (in query): integerlimitpretty (in query): stringprettyresourceVersion (in query): stringresourceVersionresourceVersionMatch (in query): stringresourceVersionMatchsendInitialEvents (in query): booleansendInitialEventstimeoutSeconds (in query): integertimeoutSecondswatch (in query): booleanwatchResponse200 (ResourceClaimTemplateList): OK401: Unauthorizedcreate create a ResourceClaimTemplateHTTP RequestPOST /apis/resource.k8s.io/v1/namespaces/{namespace}/resourceclaimtemplatesParametersnamespace (in path): string, requirednamespacebody: ResourceClaimTemplate, requireddryRun (in query): stringdryRunfieldManager (in query): stringfieldManagerfieldValidation (in query): stringfieldValidationpretty (in query): stringprettyResponse200 (ResourceClaimTemplate): OK201 (ResourceClaimTemplate): Created202 (ResourceClaimTemplate): Accepted401: Unauthorizedupdate replace the specified ResourceClaimTemplateHTTP RequestPUT /apis/resource.k8s.io/v1/namespaces/{namespace}/resourceclaimtemplates/{name}Parametersname (in path): string, requiredname of the ResourceClaimTemplatenamespace (in path): string, requirednamespacebody: ResourceClaimTemplate, requireddryRun (in query): stringdryRunfieldManager (in query): stringfieldManagerfieldValidation (in query): stringfieldValidationpretty (in query): stringprettyResponse200 (ResourceClaimTemplate): OK201 (ResourceClaimTemplate): Created401: Unauthorizedpatch partially update the specified ResourceClaimTemplateHTTP RequestPATCH /apis/resource.k8s.io/v1/namespaces/{namespace}/resourceclaimtemplates/{name}Parametersname (in path): string, requiredname of the ResourceClaimTemplatenamespace (in path): string, requirednamespacebody: Patch, requireddryRun (in query): stringdryRunfieldManager (in query): stringfieldManagerfieldValidation (in query): stringfieldValidationforce (in query): booleanforcepretty (in query): stringprettyResponse200 (ResourceClaimTemplate): OK201 (ResourceClaimTemplate): Created401: Unauthorizeddelete delete a ResourceClaimTemplateHTTP RequestDELETE /apis/resource.k8s.io/v1/namespaces/{namespace}/resourceclaimtemplates/{name}Parametersname (in path): string, requiredname of the ResourceClaimTemplatenamespace (in path): string, requirednamespacebody: DeleteOptionsdryRun (in query): stringdryRungracePeriodSeconds (in query): integergracePeriodSecondsignoreStoreReadErrorWithClusterBreakingPotential (in query): booleanignoreStoreReadErrorWithClusterBreakingPotentialpretty (in query): stringprettypropagationPolicy (in query): stringpropagationPolicyResponse200 (ResourceClaimTemplate): OK202 (ResourceClaimTemplate): Accepted401: Unauthorizeddeletecollection delete collection of ResourceClaimTemplateHTTP RequestDELETE /apis/resource.k8s.io/v1/namespaces/{namespace}/resourceclaimtemplatesParametersnamespace (in path): string, requirednamespacebody: DeleteOptionscontinue (in query): stringcontinuedryRun (in query): stringdryRunfieldSelector (in query): stringfieldSelectorgracePeriodSeconds (in query): integergracePeriodSecondsignoreStoreReadErrorWithClusterBreakingPotential (in query): booleanignoreStoreReadErrorWithClusterBreakingPotentiallabelSelector (in query): stringlabelSelectorlimit (in query): integerlimitpretty (in query): stringprettypropagationPolicy (in query): stringpropagationPolicyresourceVersion (in query): stringresourceVersionresourceVersionMatch (in query): stringresourceVersionMatchsendInitialEvents (in query): booleansendInitialEventstimeoutSeconds (in query): integertimeoutSecondsResponse200 (Status): OK401: UnauthorizedThis page is automatically generated.If you plan to report an issue with this page, mention that the page is auto-generated in your issue description. The fix may need to happen elsewhere in the Kubernetes project.

ResourceClaimTemplate


Projected VolumesThis document describes projected volumes in Kubernetes. Familiarity with volumes is suggested.IntroductionA projected volume maps several existing volume sources into the same directory.Currently, the following types of volume sources can be projected:secretdownwardAPIconfigMapserviceAccountTokenclusterTrustBundlepodCertificateAll sources are required to be in the same namespace as the Pod. For more details, see the all-in-one volume design document.Example configuration with a secret, a downwardAPI, and a configMappods/storage/projected-secret-downwardapi-configmap.yaml apiVersion: v1 kind: Pod metadata: name: volume-test spec: containers: - name: container-test image: busybox:1.28 command: ["sleep", "3600"] volumeMounts: - name: all-in-one mountPath: "/projected-volume" readOnly: true volumes: - name: all-in-one projected: sources: - secret: name: mysecret items: - key: username path: my-group/my-username - downwardAPI: items: - path: "labels" fieldRef: fieldPath: metadata.labels - path: "cpu_limit" resourceFieldRef: containerName: container-test resource: limits.cpu - configMap: name: myconfigmap items: - key: config path: my-group/my-config Example configuration: secrets with a non-default permission mode setpods/storage/projected-secrets-nondefault-permission-mode.yaml apiVersion: v1 kind: Pod metadata: name: volume-test spec: containers: - name: container-test image: busybox:1.28 command: ["sleep", "3600"] volumeMounts: - name: all-in-one mountPath: "/projected-volume" readOnly: true volumes: - name: all-in-one projected: sources: - secret: name: mysecret items: - key: username path: my-group/my-username - secret: name: mysecret2 items: - key: password path: my-group/my-password mode: 511 Each projected volume source is listed in the spec under sources. The parameters are nearly the same with two exceptions:For secrets, the secretName field has been changed to name to be consistent with ConfigMap naming.The defaultMode can only be specified at the projected level and not for each volume source. However, as illustrated above, you can explicitly set the mode for each individual projection.serviceAccountToken projected volumesYou can inject the token for the current service account into a Pod at a specified path. For example:pods/storage/projected-service-account-token.yaml apiVersion: v1 kind: Pod metadata: name: sa-token-test spec: containers: - name: container-test image: busybox:1.28 command: ["sleep", "3600"] volumeMounts: - name: token-vol mountPath: "/service-account" readOnly: true serviceAccountName: default volumes: - name: token-vol projected: sources: - serviceAccountToken: audience: api expirationSeconds: 3600 path: token The example Pod has a projected volume containing the injected service account token. Containers in this Pod can use that token to access the Kubernetes API server, authenticating with the identity of the pod's ServiceAccount. The audience field contains the intended audience of the token. A recipient of the token must identify itself with an identifier specified in the audience of the token, and otherwise should reject the token. This field is optional and it defaults to the identifier of the API server.The expirationSeconds is the expected duration of validity of the service account token. It defaults to 1 hour and must be at least 10 minutes (600 seconds). An administrator can also limit its maximum value by specifying the --service-account-max-token-expiration option for the API server. The path field specifies a relative path to the mount point of the projected volume.Note:A container using a projected volume source as a subPath volume mount will not receive updates for those volume sources.clusterTrustBundle projected volumesFEATURE STATE: Kubernetes v1.33 [beta] (enabled by default: false)Note:To use this feature in Kubernetes 1.34, you must enable support for ClusterTrustBundle objects with the ClusterTrustBundle feature gate and --runtime-config=certificates.k8s.io/v1beta1/clustertrustbundles=true kube-apiserver flag, then enable the ClusterTrustBundleProjection feature gate.The clusterTrustBundle projected volume source injects the contents of one or more ClusterTrustBundle objects as an automatically-updating file in the container filesystem.ClusterTrustBundles can be selected either by name or by signer name.To select by name, use the name field to designate a single ClusterTrustBundle object.To select by signer name, use the signerName field (and optionally the labelSelector field) to designate a set of ClusterTrustBundle objects that use the given signer name. If labelSelector is not present, then all ClusterTrustBundles for that signer are selected.The kubelet deduplicates the certificates in the selected ClusterTrustBundle objects, normalizes the PEM representations (discarding comments and headers), reorders the certificates, and writes them into the file named by path. As the set of selected ClusterTrustBundles or their content changes, kubelet keeps the file up-to-date.By default, the kubelet will prevent the pod from starting if the named ClusterTrustBundle is not found, or if signerName / labelSelector do not match any ClusterTrustBundles. If this behavior is not what you want, then set the optional field to true, and the pod will start up with an empty file at path.pods/storage/projected-clustertrustbundle.yaml apiVersion: v1 kind: Pod metadata: name: sa-ctb-name-test spec: containers: - name: container-test image: busybox command: ["sleep", "3600"] volumeMounts: - name: token-vol mountPath: "/root-certificates" readOnly: true serviceAccountName: default volumes: - name: token-vol projected: sources: - clusterTrustBundle: name: example path: example-roots.pem - clusterTrustBundle: signerName: "example.com/mysigner" labelSelector: matchLabels: version: live path: mysigner-roots.pem optional: true podCertificate projected volumesFEATURE STATE: Kubernetes v1.34 [alpha] (enabled by default: false)Note:In Kubernetes 1.34, you must enable support for Pod Certificates using the PodCertificateRequest feature gate and the --runtime-config=certificates.k8s.io/v1alpha1/podcertificaterequests=true kube-apiserver flag.The podCertificate projected volumes source securely provisions a private key and X.509 certificate chain for pod to use as client or server credentials. Kubelet will then handle refreshing the private key and certificate chain when they get close to expiration. The application just has to make sure that it reloads the file promptly when it changes, with a mechanism like inotify or polling.Each podCertificate projection supports the following configuration fields:signerName: The signer you want to issue the certificate. Note that signers may have their own access requirements, and may refuse to issue certificates to your pod.keyType: The type of private key that should be generated. Valid values are ED25519, ECDSAP256, ECDSAP384, ECDSAP521, RSA3072, and RSA4096.maxExpirationSeconds: The maximum lifetime you will accept for the certificate issued to the pod. If not set, will be defaulted to 86400 (24 hours). Must be at least 3600 (1 hour), and at most 7862400 (91 days). Kubernetes built-in signers are restricted to a max lifetime of 86400 (1 day). The signer is allowed to issue a certificate with a lifetime shorter than what you've specified.credentialBundlePath: Relative path within the projection where the credential bundle should be written. The credential bundle is a PEM-formatted file, where the first block is a "PRIVATE KEY" block that contains a PKCS#8-serialized private key, and the remaining blocks are "CERTIFICATE" blocks that comprise the certificate chain (leaf certificate and any intermediates).keyPath and certificateChainPath: Separate paths where Kubelet should write just the private key or certificate chain.Note:Most applications should prefer using credentialBundlePath unless they need the key and certificates in separate files for compatibility reasons. Kubelet uses an atomic writing strategy based on symlinks to make sure that when you open the files it projects, you read either the old content or the new content. However, if you read the key and certificate chain from separate files, Kubelet may rotate the credentials after your first read and before your second read, resulting in your application loading a mismatched key and certificate.pods/storage/projected-podcertificate.yaml # Sample Pod spec that uses a podCertificate projection to request an ED25519 # private key, a certificate from the `coolcert.example.com/foo` signer, and # write the results to `/var/run/my-x509-credentials/credentialbundle.pem`. apiVersion: v1 kind: Pod metadata: namespace: default name: podcertificate-pod spec: serviceAccountName: default containers: - image: debian name: main command: ["sleep", "infinity"] volumeMounts: - name: my-x509-credentials mountPath: /var/run/my-x509-credentials volumes: - name: my-x509-credentials projected: defaultMode: 420 sources: - podCertificate: keyType: ED25519 signerName: coolcert.example.com/foo credentialBundlePath: credentialbundle.pem SecurityContext interactionsThe proposal for file permission handling in projected service account volume enhancement introduced the projected files having the correct owner permissions set.LinuxIn Linux pods that have a projected volume and RunAsUser set in the Pod SecurityContext, the projected files have the correct ownership set including container user ownership.When all containers in a pod have the same runAsUser set in their PodSecurityContext or container SecurityContext, then the kubelet ensures that the contents of the serviceAccountToken volume are owned by that user, and the token file has its permission mode set to 0600.Note:Ephemeral containers added to a Pod after it is created do not change volume permissions that were set when the pod was created.If a Pod's serviceAccountToken volume permissions were set to 0600 because all other containers in the Pod have the same runAsUser, ephemeral containers must use the same runAsUser to be able to read the token.WindowsIn Windows pods that have a projected volume and RunAsUsername set in the Pod SecurityContext, the ownership is not enforced due to the way user accounts are managed in Windows. Windows stores and manages local user and group accounts in a database file called Security Account Manager (SAM). Each container maintains its own instance of the SAM database, to which the host has no visibility into while the container is running. Windows containers are designed to run the user mode portion of the OS in isolation from the host, hence the maintenance of a virtual SAM database. As a result, the kubelet running on the host does not have the ability to dynamically configure host file ownership for virtualized container accounts. It is recommended that if files on the host machine are to be shared with the container then they should be placed into their own volume mount outside of C:\.By default, the projected files will have the following ownership as shown for an example projected volume file:PS C:\> Get-Acl C:\var\run\secrets\kubernetes.io\serviceaccount\..2021_08_31_22_22_18.318230061\ca.crt | Format-List Path : Microsoft.PowerShell.Core\FileSystem::C:\var\run\secrets\kubernetes.io\serviceaccount\..2021_08_31_22_22_18.318230061\ca.crt Owner : BUILTIN\Administrators Group : NT AUTHORITY\SYSTEM Access : NT AUTHORITY\SYSTEM Allow FullControl BUILTIN\Administrators Allow FullControl BUILTIN\Users Allow ReadAndExecute, Synchronize Audit : Sddl : O:BAG:SYD:AI(A;ID;FA;;;SY)(A;ID;FA;;;BA)(A;ID;0x1200a9;;;BU) This implies all administrator users like ContainerAdministrator will have read, write and execute access while, non-administrator users will have read and execute access.Note:In general, granting the container access to the host is discouraged as it can open the door for potential security exploits.Creating a Windows Pod with RunAsUser in it's SecurityContext will result in the Pod being stuck at ContainerCreating forever. So it is advised to not use the Linux only RunAsUser option with Windows Pods.

Projected Volumes


Assign Pods to Nodes using Node AffinityThis page shows how to assign a Kubernetes Pod to a particular node using Node Affinity in a Kubernetes cluster.Before you beginYou need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. It is recommended to run this tutorial on a cluster with at least two nodes that are not acting as control plane hosts. If you do not already have a cluster, you can create one by using minikube or you can use one of these Kubernetes playgrounds:iximiuz LabsKillercodaKodeKloudPlay with KubernetesYour Kubernetes server must be at or later than version v1.10.To check the version, enter kubectl version.Add a label to a nodeList the nodes in your cluster, along with their labels:kubectl get nodes --show-labels The output is similar to this:NAME STATUS ROLES AGE VERSION LABELS worker0 Ready <none> 1d v1.13.0 ...,kubernetes.io/hostname=worker0 worker1 Ready <none> 1d v1.13.0 ...,kubernetes.io/hostname=worker1 worker2 Ready <none> 1d v1.13.0 ...,kubernetes.io/hostname=worker2 Choose one of your nodes, and add a label to it:kubectl label nodes <your-node-name> disktype=ssd where <your-node-name> is the name of your chosen node.Verify that your chosen node has a disktype=ssd label:kubectl get nodes --show-labels The output is similar to this:NAME STATUS ROLES AGE VERSION LABELS worker0 Ready <none> 1d v1.13.0 ...,disktype=ssd,kubernetes.io/hostname=worker0 worker1 Ready <none> 1d v1.13.0 ...,kubernetes.io/hostname=worker1 worker2 Ready <none> 1d v1.13.0 ...,kubernetes.io/hostname=worker2 In the preceding output, you can see that the worker0 node has a disktype=ssd label.Schedule a Pod using required node affinityThis manifest describes a Pod that has a requiredDuringSchedulingIgnoredDuringExecution node affinity,disktype: ssd. This means that the pod will get scheduled only on a node that has a disktype=ssd label.pods/pod-nginx-required-affinity.yaml apiVersion: v1 kind: Pod metadata: name: nginx spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: disktype operator: In values: - ssd containers: - name: nginx image: nginx imagePullPolicy: IfNotPresent Apply the manifest to create a Pod that is scheduled onto your chosen node:kubectl apply -f https://k8s.io/examples/pods/pod-nginx-required-affinity.yaml Verify that the pod is running on your chosen node:kubectl get pods --output=wide The output is similar to this:NAME READY STATUS RESTARTS AGE IP NODE nginx 1/1 Running 0 13s 10.200.0.4 worker0 Schedule a Pod using preferred node affinityThis manifest describes a Pod that has a preferredDuringSchedulingIgnoredDuringExecution node affinity,disktype: ssd. This means that the pod will prefer a node that has a disktype=ssd label.pods/pod-nginx-preferred-affinity.yaml apiVersion: v1 kind: Pod metadata: name: nginx spec: affinity: nodeAffinity: preferredDuringSchedulingIgnoredDuringExecution: - weight: 1 preference: matchExpressions: - key: disktype operator: In values: - ssd containers: - name: nginx image: nginx imagePullPolicy: IfNotPresent Apply the manifest to create a Pod that is scheduled onto your chosen node:kubectl apply -f https://k8s.io/examples/pods/pod-nginx-preferred-affinity.yaml Verify that the pod is running on your chosen node:kubectl get pods --output=wide The output is similar to this:NAME READY STATUS RESTARTS AGE IP NODE nginx 1/1 Running 0 13s 10.200.0.4 worker0 What's nextLearn more about Node Affinity.


Restrict a Container's Access to Resources with AppArmorFEATURE STATE: Kubernetes v1.31 [stable] (enabled by default: true)This page shows you how to load AppArmor profiles on your nodes and enforce those profiles in Pods. To learn more about how Kubernetes can confine Pods using AppArmor, see Linux kernel security constraints for Pods and containers.ObjectivesSee an example of how to load a profile on a NodeLearn how to enforce the profile on a PodLearn how to check that the profile is loadedSee what happens when a profile is violatedSee what happens when a profile cannot be loadedBefore you beginAppArmor is an optional kernel module and Kubernetes feature, so verify it is supported on your Nodes before proceeding:AppArmor kernel module is enabled -- For the Linux kernel to enforce an AppArmor profile, the AppArmor kernel module must be installed and enabled. Several distributions enable the module by default, such as Ubuntu and SUSE, and many others provide optional support. To check whether the module is enabled, check the /sys/module/apparmor/parameters/enabled file:cat /sys/module/apparmor/parameters/enabled Y The kubelet verifies that AppArmor is enabled on the host before admitting a pod with AppArmor explicitly configured.Container runtime supports AppArmor -- All common Kubernetes-supported container runtimes should support AppArmor, including containerd and CRI-O. Please refer to the corresponding runtime documentation and verify that the cluster fulfills the requirements to use AppArmor.Profile is loaded -- AppArmor is applied to a Pod by specifying an AppArmor profile that each container should be run with. If any of the specified profiles are not loaded in the kernel, the kubelet will reject the Pod. You can view which profiles are loaded on a node by checking the /sys/kernel/security/apparmor/profiles file. For example:ssh gke-test-default-pool-239f5d02-gyn2 "sudo cat /sys/kernel/security/apparmor/profiles | sort" apparmor-test-deny-write (enforce) apparmor-test-audit-write (enforce) docker-default (enforce) k8s-nginx (enforce) For more details on loading profiles on nodes, see Setting up nodes with profiles.Securing a PodNote:Prior to Kubernetes v1.30, AppArmor was specified through annotations. Use the documentation version selector to view the documentation with this deprecated API.AppArmor profiles can be specified at the pod level or container level. The container AppArmor profile takes precedence over the pod profile.securityContext: appArmorProfile: type: <profile_type> Where <profile_type> is one of:RuntimeDefault to use the runtime's default profileLocalhost to use a profile loaded on the host (see below)Unconfined to run without AppArmorSee Specifying AppArmor Confinement for full details on the AppArmor profile API.To verify that the profile was applied, you can check that the container's root process is running with the correct profile by examining its proc attr:kubectl exec <pod_name> -- cat /proc/1/attr/current The output should look something like this:cri-containerd.apparmor.d (enforce) ExampleThis example assumes you have already set up a cluster with AppArmor support.First, load the profile you want to use onto your Nodes. This profile blocks all file write operations:#include <tunables/global> profile k8s-apparmor-example-deny-write flags=(attach_disconnected) { #include <abstractions/base> file, # Deny all file writes. deny /** w, } The profile needs to be loaded onto all nodes, since you don't know where the pod will be scheduled. For this example you can use SSH to install the profiles, but other approaches are discussed in Setting up nodes with profiles.# This example assumes that node names match host names, and are reachable via SSH. NODES=($( kubectl get node -o jsonpath='{.items[*].status.addresses[?(.type == "Hostname")].address}' )) for NODE in ${NODES[*]}; do ssh $NODE 'sudo apparmor_parser -q <<EOF #include <tunables/global> profile k8s-apparmor-example-deny-write flags=(attach_disconnected) { #include <abstractions/base> file, # Deny all file writes. deny /** w, } EOF' done Next, run a simple "Hello AppArmor" Pod with the deny-write profile:pods/security/hello-apparmor.yaml apiVersion: v1 kind: Pod metadata: name: hello-apparmor spec: securityContext: appArmorProfile: type: Localhost localhostProfile: k8s-apparmor-example-deny-write containers: - name: hello image: busybox:1.28 command: [ "sh", "-c", "echo 'Hello AppArmor!' && sleep 1h" ] kubectl create -f hello-apparmor.yaml You can verify that the container is actually running with that profile by checking /proc/1/attr/current:kubectl exec hello-apparmor -- cat /proc/1/attr/current The output should be:k8s-apparmor-example-deny-write (enforce) Finally, you can see what happens if you violate the profile by writing to a file:kubectl exec hello-apparmor -- touch /tmp/test touch: /tmp/test: Permission denied error: error executing remote command: command terminated with non-zero exit code: Error executing in Docker Container: 1 To wrap up, see what happens if you try to specify a profile that hasn't been loaded:kubectl create -f /dev/stdin <<EOF apiVersion: v1 kind: Pod metadata: name: hello-apparmor-2 spec: securityContext: appArmorProfile: type: Localhost localhostProfile: k8s-apparmor-example-allow-write containers: - name: hello image: busybox:1.28 command: [ "sh", "-c", "echo 'Hello AppArmor!' && sleep 1h" ] EOF pod/hello-apparmor-2 created Although the Pod was created successfully, further examination will show that it is stuck in pending:kubectl describe pod hello-apparmor-2 Name: hello-apparmor-2 Namespace: default Node: gke-test-default-pool-239f5d02-x1kf/10.128.0.27 Start Time: Tue, 30 Aug 2016 17:58:56 -0700 Labels: <none> Annotations: container.apparmor.security.beta.kubernetes.io/hello=localhost/k8s-apparmor-example-allow-write Status: Pending ... Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 10s default-scheduler Successfully assigned default/hello-apparmor to gke-test-default-pool-239f5d02-x1kf Normal Pulled 8s kubelet Successfully pulled image "busybox:1.28" in 370.157088ms (370.172701ms including waiting) Normal Pulling 7s (x2 over 9s) kubelet Pulling image "busybox:1.28" Warning Failed 7s (x2 over 8s) kubelet Error: failed to get container spec opts: failed to generate apparmor spec opts: apparmor profile not found k8s-apparmor-example-allow-write Normal Pulled 7s kubelet Successfully pulled image "busybox:1.28" in 90.980331ms (91.005869ms including waiting) An Event provides the error message with the reason, the specific wording is runtime-dependent: Warning Failed 7s (x2 over 8s) kubelet Error: failed to get container spec opts: failed to generate apparmor spec opts: apparmor profile not found AdministrationSetting up Nodes with profilesKubernetes 1.34 does not provide any built-in mechanisms for loading AppArmor profiles onto Nodes. Profiles can be loaded through custom infrastructure or tools like the Kubernetes Security Profiles Operator.The scheduler is not aware of which profiles are loaded onto which Node, so the full set of profiles must be loaded onto every Node. An alternative approach is to add a Node label for each profile (or class of profiles) on the Node, and use a node selector to ensure the Pod is run on a Node with the required profile.Authoring ProfilesGetting AppArmor profiles specified correctly can be a tricky business. Fortunately there are some tools to help with that:aa-genprof and aa-logprof generate profile rules by monitoring an application's activity and logs, and admitting the actions it takes. Further instructions are provided by the AppArmor documentation.bane is an AppArmor profile generator for Docker that uses a simplified profile language.To debug problems with AppArmor, you can check the system logs to see what, specifically, was denied. AppArmor logs verbose messages to dmesg, and errors can usually be found in the system logs or through journalctl. More information is provided in AppArmor failures.Specifying AppArmor confinementCaution:Prior to Kubernetes v1.30, AppArmor was specified through annotations. Use the documentation version selector to view the documentation with this deprecated API.AppArmor profile within security contextYou can specify the appArmorProfile on either a container's securityContext or on a Pod's securityContext. If the profile is set at the pod level, it will be used as the default profile for all containers in the pod (including init, sidecar, and ephemeral containers). If both a pod & container AppArmor profile are set, the container's profile will be used.An AppArmor profile has 2 fields:type (required) - indicates which kind of AppArmor profile will be applied. Valid options are:Localhosta profile pre-loaded on the node (specified by localhostProfile).RuntimeDefaultthe container runtime's default profile.Unconfinedno AppArmor enforcement.localhostProfile - The name of a profile loaded on the node that should be used. The profile must be preconfigured on the node to work. This option must be provided if and only if the type is Localhost.What's nextAdditional resources:Quick guide to the AppArmor profile languageAppArmor core policy reference