# Declarative Management of Kubernetes Objects Using Configuration Files

Declarative Management of Kubernetes Objects Using Configuration FilesKubernetes objects can be created, updated, and deleted by storing multiple object configuration files in a directory and using kubectl apply to recursively create and update those objects as needed. This method retains writes made to live objects without merging the changes back into the object configuration files. kubectl diff also gives you a preview of what changes apply will make.Before you beginInstall kubectl.You need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. It is recommended to run this tutorial on a cluster with at least two nodes that are not acting as control plane hosts. If you do not already have a cluster, you can create one by using minikube or you can use one of these Kubernetes playgrounds:iximiuz LabsKillercodaKodeKloudPlay with KubernetesTo check the version, enter kubectl version.Trade-offsThe kubectl tool supports three kinds of object management:Imperative commandsImperative object configurationDeclarative object configurationSee Kubernetes Object Management for a discussion of the advantages and disadvantage of each kind of object management.OverviewDeclarative object configuration requires a firm understanding of the Kubernetes object definitions and configuration. Read and complete the following documents if you have not already:Managing Kubernetes Objects Using Imperative CommandsImperative Management of Kubernetes Objects Using Configuration FilesFollowing are definitions for terms used in this document:object configuration file / configuration file: A file that defines the configuration for a Kubernetes object. This topic shows how to pass configuration files to kubectl apply. Configuration files are typically stored in source control, such as Git.live object configuration / live configuration: The live configuration values of an object, as observed by the Kubernetes cluster. These are kept in the Kubernetes cluster storage, typically etcd.declarative configuration writer / declarative writer: A person or software component that makes updates to a live object. The live writers referred to in this topic make changes to object configuration files and run kubectl apply to write the changes.How to create objectsUse kubectl apply to create all objects, except those that already exist, defined by configuration files in a specified directory:kubectl apply -f <directory> This sets the kubectl.kubernetes.io/last-applied-configuration: '{...}' annotation on each object. The annotation contains the contents of the object configuration file that was used to create the object.Note:Add the -R flag to recursively process directories.Here's an example of an object configuration file:application/simple_deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment spec: selector: matchLabels: app: nginx minReadySeconds: 5 template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx:1.14.2 ports: - containerPort: 80 Run kubectl diff to print the object that will be created:kubectl diff -f https://k8s.io/examples/application/simple_deployment.yaml Note:diff uses server-side dry-run, which needs to be enabled on kube-apiserver.Since diff performs a server-side apply request in dry-run mode, it requires granting PATCH, CREATE, and UPDATE permissions. See Dry-Run Authorization for details.Create the object using kubectl apply:kubectl apply -f https://k8s.io/examples/application/simple_deployment.yaml Print the live configuration using kubectl get:kubectl get -f https://k8s.io/examples/application/simple_deployment.yaml -o yaml The output shows that the kubectl.kubernetes.io/last-applied-configuration annotation was written to the live configuration, and it matches the configuration file:kind: Deployment metadata: annotations: # ... # This is the json representation of simple_deployment.yaml # It was written by kubectl apply when the object was created kubectl.kubernetes.io/last-applied-configuration: | {"apiVersion":"apps/v1","kind":"Deployment", "metadata":{"annotations":{},"name":"nginx-deployment","namespace":"default"}, "spec":{"minReadySeconds":5,"selector":{"matchLabels":{"app":nginx}},"template":{"metadata":{"labels":{"app":"nginx"}}, "spec":{"containers":[{"image":"nginx:1.14.2","name":"nginx", "ports":[{"containerPort":80}]}]}}}} # ... spec: # ... minReadySeconds: 5 selector: matchLabels: # ... app: nginx template: metadata: # ... labels: app: nginx spec: containers: - image: nginx:1.14.2 # ... name: nginx ports: - containerPort: 80 # ... # ... # ... # ... How to update objectsYou can also use kubectl apply to update all objects defined in a directory, even if those objects already exist. This approach accomplishes the following:Sets fields that appear in the configuration file in the live configuration.Clears fields removed from the configuration file in the live configuration.kubectl diff -f <directory> kubectl apply -f <directory> Note:Add the -R flag to recursively process directories.Here's an example configuration file:application/simple_deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment spec: selector: matchLabels: app: nginx minReadySeconds: 5 template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx:1.14.2 ports: - containerPort: 80 Create the object using kubectl apply:kubectl apply -f https://k8s.io/examples/application/simple_deployment.yaml Note:For purposes of illustration, the preceding command refers to a single configuration file instead of a directory.Print the live configuration using kubectl get:kubectl get -f https://k8s.io/examples/application/simple_deployment.yaml -o yaml The output shows that the kubectl.kubernetes.io/last-applied-configuration annotation was written to the live configuration, and it matches the configuration file:kind: Deployment metadata: annotations: # ... # This is the json representation of simple_deployment.yaml # It was written by kubectl apply when the object was created kubectl.kubernetes.io/last-applied-configuration: | {"apiVersion":"apps/v1","kind":"Deployment", "metadata":{"annotations":{},"name":"nginx-deployment","namespace":"default"}, "spec":{"minReadySeconds":5,"selector":{"matchLabels":{"app":nginx}},"template":{"metadata":{"labels":{"app":"nginx"}}, "spec":{"containers":[{"image":"nginx:1.14.2","name":"nginx", "ports":[{"containerPort":80}]}]}}}} # ... spec: # ... minReadySeconds: 5 selector: matchLabels: # ... app: nginx template: metadata: # ... labels: app: nginx spec: containers: - image: nginx:1.14.2 # ... name: nginx ports: - containerPort: 80 # ... # ... # ... # ... Directly update the replicas field in the live configuration by using kubectl scale. This does not use kubectl apply:kubectl scale deployment/nginx-deployment --replicas=2 Print the live configuration using kubectl get:kubectl get deployment nginx-deployment -o yaml The output shows that the replicas field has been set to 2, and the last-applied-configuration annotation does not contain a replicas field:apiVersion: apps/v1 kind: Deployment metadata: annotations: # ... # note that the annotation does not contain replicas # because it was not updated through apply kubectl.kubernetes.io/last-applied-configuration: | {"apiVersion":"apps/v1","kind":"Deployment", "metadata":{"annotations":{},"name":"nginx-deployment","namespace":"default"}, "spec":{"minReadySeconds":5,"selector":{"matchLabels":{"app":nginx}},"template":{"metadata":{"labels":{"app":"nginx"}}, "spec":{"containers":[{"image":"nginx:1.14.2","name":"nginx", "ports":[{"containerPort":80}]}]}}}} # ... spec: replicas: 2 # written by scale # ... minReadySeconds: 5 selector: matchLabels: # ... app: nginx template: metadata: # ... labels: app: nginx spec: containers: - image: nginx:1.14.2 # ... name: nginx ports: - containerPort: 80 # ... Update the simple_deployment.yaml configuration file to change the image from nginx:1.14.2 to nginx:1.16.1, and delete the minReadySeconds field:application/update_deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment spec: selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx:1.16.1 # update the image ports: - containerPort: 80 Apply the changes made to the configuration file:kubectl diff -f https://k8s.io/examples/application/update_deployment.yaml kubectl apply -f https://k8s.io/examples/application/update_deployment.yaml Print the live configuration using kubectl get:kubectl get -f https://k8s.io/examples/application/update_deployment.yaml -o yaml The output shows the following changes to the live configuration:The replicas field retains the value of 2 set by kubectl scale. This is possible because it is omitted from the configuration file.The image field has been updated to nginx:1.16.1 from nginx:1.14.2.The last-applied-configuration annotation has been updated with the new image.The minReadySeconds field has been cleared.The last-applied-configuration annotation no longer contains the minReadySeconds field.apiVersion: apps/v1 kind: Deployment metadata: annotations: # ... # The annotation contains the updated image to nginx 1.16.1, # but does not contain the updated replicas to 2 kubectl.kubernetes.io/last-applied-configuration: | {"apiVersion":"apps/v1","kind":"Deployment", "metadata":{"annotations":{},"name":"nginx-deployment","namespace":"default"}, "spec":{"selector":{"matchLabels":{"app":nginx}},"template":{"metadata":{"labels":{"app":"nginx"}}, "spec":{"containers":[{"image":"nginx:1.16.1","name":"nginx", "ports":[{"containerPort":80}]}]}}}} # ... spec: replicas: 2 # Set by `kubectl scale`. Ignored by `kubectl apply`. # minReadySeconds cleared by `kubectl apply` # ... selector: matchLabels: # ... app: nginx template: metadata: # ... labels: app: nginx spec: containers: - image: nginx:1.16.1 # Set by `kubectl apply` # ... name: nginx ports: - containerPort: 80 # ... # ... # ... # ... Warning:Mixing kubectl apply with the imperative object configuration commands create and replace is not supported. This is because create and replace do not retain the kubectl.kubernetes.io/last-applied-configuration that kubectl apply uses to compute updates.How to delete objectsThere are two approaches to delete objects managed by kubectl apply.Recommended: kubectl delete -f <filename>Manually deleting objects using the imperative command is the recommended approach, as it is more explicit about what is being deleted, and less likely to result in the user deleting something unintentionally:kubectl delete -f <filename> Alternative: kubectl apply -f <directory> --pruneAs an alternative to kubectl delete, you can use kubectl apply to identify objects to be deleted after their manifests have been removed from a directory in the local filesystem.In Kubernetes 1.34, there are two pruning modes available in kubectl apply:Allowlist-based pruning: This mode has existed since kubectl v1.5 but is still in alpha due to usability, correctness and performance issues with its design. The ApplySet-based mode is designed to replace it.ApplySet-based pruning: An apply set is a server-side object (by default, a Secret) that kubectl can use to accurately and efficiently track set membership across apply operations. This mode was introduced in alpha in kubectl v1.27 as a replacement for allowlist-based pruning.FEATURE STATE: Kubernetes v1.5 [alpha]Warning:Take care when using --prune with kubectl apply in allow list mode. Which objects are pruned depends on the values of the --prune-allowlist, --selector and --namespace flags, and relies on dynamic discovery of the objects in scope. Especially if flag values are changed between invocations, this can lead to objects being unexpectedly deleted or retained.To use allowlist-based pruning, add the following flags to your kubectl apply invocation:--prune: Delete previously applied objects that are not in the set passed to the current invocation.--prune-allowlist: A list of group-version-kinds (GVKs) to consider for pruning. This flag is optional but strongly encouraged, as its default value is a partial list of both namespaced and cluster-scoped types, which can lead to surprising results.--selector/-l: Use a label selector to constrain the set of objects selected for pruning. This flag is optional but strongly encouraged.--all: use instead of --selector/-l to explicitly select all previously applied objects of the allowlisted types.Allowlist-based pruning queries the API server for all objects of the allowlisted GVKs that match the given labels (if any), and attempts to match the returned live object configurations against the object manifest files. If an object matches the query, and it does not have a manifest in the directory, and it has a kubectl.kubernetes.io/last-applied-configuration annotation, it is deleted.kubectl apply -f <directory> --prune -l <labels> --prune-allowlist=<gvk-list> Warning:Apply with prune should only be run against the root directory containing the object manifests. Running against sub-directories can cause objects to be unintentionally deleted if they were previously applied, have the labels given (if any), and do not appear in the subdirectory.FEATURE STATE: Kubernetes v1.27 [alpha]Caution:kubectl apply --prune --applyset is in alpha, and backwards incompatible changes might be introduced in subsequent releases.To use ApplySet-based pruning, set the KUBECTL_APPLYSET=true environment variable, and add the following flags to your kubectl apply invocation:--prune: Delete previously applied objects that are not in the set passed to the current invocation.--applyset: The name of an object that kubectl can use to accurately and efficiently track set membership across apply operations.KUBECTL_APPLYSET=true kubectl apply -f <directory> --prune --applyset=<name> By default, the type of the ApplySet parent object used is a Secret. However, ConfigMaps can also be used in the format: --applyset=configmaps/<name>. When using a Secret or ConfigMap, kubectl will create the object if it does not already exist.It is also possible to use custom resources as ApplySet parent objects. To enable this, label the Custom Resource Definition (CRD) that defines the resource you want to use with the following: applyset.kubernetes.io/is-parent-type: true. Then, create the object you want to use as an ApplySet parent (kubectl does not do this automatically for custom resources). Finally, refer to that object in the applyset flag as follows: --applyset=<resource>.<group>/<name> (for example, widgets.custom.example.com/widget-name).With ApplySet-based pruning, kubectl adds the applyset.kubernetes.io/part-of=<parentID> label to each object in the set before they are sent to the server. For performance reasons, it also collects the list of resource types and namespaces that the set contains and adds these in annotations on the live parent object. Finally, at the end of the apply operation, it queries the API server for objects of those types in those namespaces (or in the cluster scope, as applicable) that belong to the set, as defined by the applyset.kubernetes.io/part-of=<parentID> label.Caveats and restrictions:Each object may be a member of at most one set.The --namespace flag is required when using any namespaced parent, including the default Secret. This means that ApplySets spanning multiple namespaces must use a cluster-scoped custom resource as the parent object.To safely use ApplySet-based pruning with multiple directories, use a unique ApplySet name for each.How to view an objectYou can use kubectl get with -o yaml to view the configuration of a live object:kubectl get -f <filename|url> -o yaml How apply calculates differences and merges changesCaution:A patch is an update operation that is scoped to specific fields of an object instead of the entire object. This enables updating only a specific set of fields on an object without reading the object first.When kubectl apply updates the live configuration for an object, it does so by sending a patch request to the API server. The patch defines updates scoped to specific fields of the live object configuration. The kubectl apply command calculates this patch request using the configuration file, the live configuration, and the last-applied-configuration annotation stored in the live configuration.Merge patch calculationThe kubectl apply command writes the contents of the configuration file to the kubectl.kubernetes.io/last-applied-configuration annotation. This is used to identify fields that have been removed from the configuration file and need to be cleared from the live configuration. Here are the steps used to calculate which fields should be deleted or set:Calculate the fields to delete. These are the fields present in last-applied-configuration and missing from the configuration file.Calculate the fields to add or set. These are the fields present in the configuration file whose values don't match the live configuration.Here's an example. Suppose this is the configuration file for a Deployment object:application/update_deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment spec: selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx:1.16.1 # update the image ports: - containerPort: 80 Also, suppose this is the live configuration for the same Deployment object:apiVersion: apps/v1 kind: Deployment metadata: annotations: # ... # note that the annotation does not contain replicas # because it was not updated through apply kubectl.kubernetes.io/last-applied-configuration: | {"apiVersion":"apps/v1","kind":"Deployment", "metadata":{"annotations":{},"name":"nginx-deployment","namespace":"default"}, "spec":{"minReadySeconds":5,"selector":{"matchLabels":{"app":nginx}},"template":{"metadata":{"labels":{"app":"nginx"}}, "spec":{"containers":[{"image":"nginx:1.14.2","name":"nginx", "ports":[{"containerPort":80}]}]}}}} # ... spec: replicas: 2 # written by scale # ... minReadySeconds: 5 selector: matchLabels: # ... app: nginx template: metadata: # ... labels: app: nginx spec: containers: - image: nginx:1.14.2 # ... name: nginx ports: - containerPort: 80 # ... Here are the merge calculations that would be performed by kubectl apply:Calculate the fields to delete by reading values from last-applied-configuration and comparing them to values in the configuration file. Clear fields explicitly set to null in the local object configuration file regardless of whether they appear in the last-applied-configuration. In this example, minReadySeconds appears in the last-applied-configuration annotation, but does not appear in the configuration file. Action: Clear minReadySeconds from the live configuration.Calculate the fields to set by reading values from the configuration file and comparing them to values in the live configuration. In this example, the value of image in the configuration file does not match the value in the live configuration. Action: Set the value of image in the live configuration.Set the last-applied-configuration annotation to match the value of the configuration file.Merge the results from 1, 2, 3 into a single patch request to the API server.Here is the live configuration that is the result of the merge:apiVersion: apps/v1 kind: Deployment metadata: annotations: # ... # The annotation contains the updated image to nginx 1.16.1, # but does not contain the updated replicas to 2 kubectl.kubernetes.io/last-applied-configuration: | {"apiVersion":"apps/v1","kind":"Deployment", "metadata":{"annotations":{},"name":"nginx-deployment","namespace":"default"}, "spec":{"selector":{"matchLabels":{"app":nginx}},"template":{"metadata":{"labels":{"app":"nginx"}}, "spec":{"containers":[{"image":"nginx:1.16.1","name":"nginx", "ports":[{"containerPort":80}]}]}}}} # ... spec: selector: matchLabels: # ... app: nginx replicas: 2 # Set by `kubectl scale`. Ignored by `kubectl apply`. # minReadySeconds cleared by `kubectl apply` # ... template: metadata: # ... labels: app: nginx spec: containers: - image: nginx:1.16.1 # Set by `kubectl apply` # ... name: nginx ports: - containerPort: 80 # ... # ... # ... # ... How different types of fields are mergedHow a particular field in a configuration file is merged with the live configuration depends on the type of the field. There are several types of fields:primitive: A field of type string, integer, or boolean. For example, image and replicas are primitive fields. Action: Replace.map, also called object: A field of type map or a complex type that contains subfields. For example, labels, annotations,spec and metadata are all maps. Action: Merge elements or subfields.list: A field containing a list of items that can be either primitive types or maps. For example, containers, ports, and args are lists. Action: Varies.When kubectl apply updates a map or list field, it typically does not replace the entire field, but instead updates the individual subelements. For instance, when merging the spec on a Deployment, the entire spec is not replaced. Instead the subfields of spec, such as replicas, are compared and merged.Merging changes to primitive fieldsPrimitive fields are replaced or cleared.Note:- is used for "not applicable" because the value is not used.Field in object configuration fileField in live object configurationField in last-applied-configurationActionYesYes-Set live to configuration file value.YesNo-Set live to local configuration.No-YesClear from live configuration.No-NoDo nothing. Keep live value.Merging changes to map fieldsFields that represent maps are merged by comparing each of the subfields or elements of the map:Note:- is used for "not applicable" because the value is not used.Key in object configuration fileKey in live object configurationField in last-applied-configurationActionYesYes-Compare sub fields values.YesNo-Set live to local configuration.No-YesDelete from live configuration.No-NoDo nothing. Keep live value.Merging changes for fields of type listMerging changes to a list uses one of three strategies:Replace the list if all its elements are primitives.Merge individual elements in a list of complex elements.Merge a list of primitive elements.The choice of strategy is made on a per-field basis.Replace the list if all its elements are primitivesTreat the list the same as a primitive field. Replace or delete the entire list. This preserves ordering.Example: Use kubectl apply to update the args field of a Container in a Pod. This sets the value of args in the live configuration to the value in the configuration file. Any args elements that had previously been added to the live configuration are lost. The order of the args elements defined in the configuration file is retained in the live configuration.# last-applied-configuration value args: ["a", "b"] # configuration file value args: ["a", "c"] # live configuration args: ["a", "b", "d"] # result after merge args: ["a", "c"] Explanation: The merge used the configuration file value as the new list value.Merge individual elements of a list of complex elements:Treat the list as a map, and treat a specific field of each element as a key. Add, delete, or update individual elements. This does not preserve ordering.This merge strategy uses a special tag on each field called a patchMergeKey. The patchMergeKey is defined for each field in the Kubernetes source code: types.go When merging a list of maps, the field specified as the patchMergeKey for a given element is used like a map key for that element.Example: Use kubectl apply to update the containers field of a PodSpec. This merges the list as though it was a map where each element is keyed by name.# last-applied-configuration value containers: - name: nginx image: nginx:1.16 - name: nginx-helper-a # key: nginx-helper-a; will be deleted in result image: helper:1.3 - name: nginx-helper-b # key: nginx-helper-b; will be retained image: helper:1.3 # configuration file value containers: - name: nginx image: nginx:1.16 - name: nginx-helper-b image: helper:1.3 - name: nginx-helper-c # key: nginx-helper-c; will be added in result image: helper:1.3 # live configuration containers: - name: nginx image: nginx:1.16 - name: nginx-helper-a image: helper:1.3 - name: nginx-helper-b image: helper:1.3 args: ["run"] # Field will be retained - name: nginx-helper-d # key: nginx-helper-d; will be retained image: helper:1.3 # result after merge containers: - name: nginx image: nginx:1.16 # Element nginx-helper-a was deleted - name: nginx-helper-b image: helper:1.3 args: ["run"] # Field was retained - name: nginx-helper-c # Element was added image: helper:1.3 - name: nginx-helper-d # Element was ignored image: helper:1.3 Explanation:The container named "nginx-helper-a" was deleted because no container named "nginx-helper-a" appeared in the configuration file.The container named "nginx-helper-b" retained the changes to args in the live configuration. kubectl apply was able to identify that "nginx-helper-b" in the live configuration was the same "nginx-helper-b" as in the configuration file, even though their fields had different values (no args in the configuration file). This is because the patchMergeKey field value (name) was identical in both.The container named "nginx-helper-c" was added because no container with that name appeared in the live configuration, but one with that name appeared in the configuration file.The container named "nginx-helper-d" was retained because no element with that name appeared in the last-applied-configuration.Merge a list of primitive elementsAs of Kubernetes 1.5, merging lists of primitive elements is not supported.Note:Which of the above strategies is chosen for a given field is controlled by the patchStrategy tag in types.go If no patchStrategy is specified for a field of type list, then the list is replaced.Default field valuesThe API server sets certain fields to default values in the live configuration if they are not specified when the object is created.Here's a configuration file for a Deployment. The file does not specify strategy:application/simple_deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment spec: selector: matchLabels: app: nginx minReadySeconds: 5 template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx:1.14.2 ports: - containerPort: 80 Create the object using kubectl apply:kubectl apply -f https://k8s.io/examples/application/simple_deployment.yaml Print the live configuration using kubectl get:kubectl get -f https://k8s.io/examples/application/simple_deployment.yaml -o yaml The output shows that the API server set several fields to default values in the live configuration. These fields were not specified in the configuration file.apiVersion: apps/v1 kind: Deployment # ... spec: selector: matchLabels: app: nginx minReadySeconds: 5 replicas: 1 # defaulted by apiserver strategy: rollingUpdate: # defaulted by apiserver - derived from strategy.type maxSurge: 1 maxUnavailable: 1 type: RollingUpdate # defaulted by apiserver template: metadata: creationTimestamp: null labels: app: nginx spec: containers: - image: nginx:1.14.2 imagePullPolicy: IfNotPresent # defaulted by apiserver name: nginx ports: - containerPort: 80 protocol: TCP # defaulted by apiserver resources: {} # defaulted by apiserver terminationMessagePath: /dev/termination-log # defaulted by apiserver dnsPolicy: ClusterFirst # defaulted by apiserver restartPolicy: Always # defaulted by apiserver securityContext: {} # defaulted by apiserver terminationGracePeriodSeconds: 30 # defaulted by apiserver # ... In a patch request, defaulted fields are not re-defaulted unless they are explicitly cleared as part of a patch request. This can cause unexpected behavior for fields that are defaulted based on the values of other fields. When the other fields are later changed, the values defaulted from them will not be updated unless they are explicitly cleared.For this reason, it is recommended that certain fields defaulted by the server are explicitly defined in the configuration file, even if the desired values match the server defaults. This makes it easier to recognize conflicting values that will not be re-defaulted by the server.Example:# last-applied-configuration spec: template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx:1.14.2 ports: - containerPort: 80 # configuration file spec: strategy: type: Recreate # updated value template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx:1.14.2 ports: - containerPort: 80 # live configuration spec: strategy: type: RollingUpdate # defaulted value rollingUpdate: # defaulted value derived from type maxSurge : 1 maxUnavailable: 1 template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx:1.14.2 ports: - containerPort: 80 # result after merge - ERROR! spec: strategy: type: Recreate # updated value: incompatible with rollingUpdate rollingUpdate: # defaulted value: incompatible with "type: Recreate" maxSurge : 1 maxUnavailable: 1 template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx:1.14.2 ports: - containerPort: 80 Explanation:The user creates a Deployment without defining strategy.type.The server defaults strategy.type to RollingUpdate and defaults the strategy.rollingUpdate values.The user changes strategy.type to Recreate. The strategy.rollingUpdate values remain at their defaulted values, though the server expects them to be cleared. If the strategy.rollingUpdate values had been defined initially in the configuration file, it would have been more clear that they needed to be deleted.Apply fails because strategy.rollingUpdate is not cleared. The strategy.rollingupdate field cannot be defined with a strategy.type of Recreate.Recommendation: These fields should be explicitly defined in the object configuration file:Selectors and PodTemplate labels on workloads, such as Deployment, StatefulSet, Job, DaemonSet, ReplicaSet, and ReplicationControllerDeployment rollout strategyHow to clear server-defaulted fields or fields set by other writersFields that do not appear in the configuration file can be cleared by setting their values to null and then applying the configuration file. For fields defaulted by the server, this triggers re-defaulting the values.How to change ownership of a field between the configuration file and direct imperative writersThese are the only methods you should use to change an individual object field:Use kubectl apply.Write directly to the live configuration without modifying the configuration file: for example, use kubectl scale.Changing the owner from a direct imperative writer to a configuration fileAdd the field to the configuration file. For the field, discontinue direct updates to the live configuration that do not go through kubectl apply.Changing the owner from a configuration file to a direct imperative writerAs of Kubernetes 1.5, changing ownership of a field from a configuration file to an imperative writer requires manual steps:Remove the field from the configuration file.Remove the field from the kubectl.kubernetes.io/last-applied-configuration annotation on the live object.Changing management methodsKubernetes objects should be managed using only one method at a time. Switching from one method to another is possible, but is a manual process.Note:It is OK to use imperative deletion with declarative management.Migrating from imperative command management to declarative object configurationMigrating from imperative command management to declarative object configuration involves several manual steps:Export the live object to a local configuration file:kubectl get <kind>/<name> -o yaml > <kind>_<name>.yaml Manually remove the status field from the configuration file.Note:This step is optional, as kubectl apply does not update the status field even if it is present in the configuration file.Set the kubectl.kubernetes.io/last-applied-configuration annotation on the object:kubectl replace --save-config -f <kind>_<name>.yaml Change processes to use kubectl apply for managing the object exclusively.Migrating from imperative object configuration to declarative object configurationSet the kubectl.kubernetes.io/last-applied-configuration annotation on the object:kubectl replace --save-config -f <kind>_<name>.yaml Change processes to use kubectl apply for managing the object exclusively.Defining controller selectors and PodTemplate labelsWarning:Updating selectors on controllers is strongly discouraged.The recommended approach is to define a single, immutable PodTemplate label used only by the controller selector with no other semantic meaning.Example:selector: matchLabels: controller-selector: "apps/v1/deployment/nginx" template: metadata: labels: controller-selector: "apps/v1/deployment/nginx" What's nextManaging Kubernetes Objects Using Imperative CommandsImperative Management of Kubernetes Objects Using Configuration FilesKubectl Command ReferenceKubernetes API Reference


# DeleteOptions

DeleteOptionsDeleteOptions may be provided when deleting an API object.import "k8s.io/apimachinery/pkg/apis/meta/v1"DeleteOptions may be provided when deleting an API object.apiVersion (string)APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resourcesdryRun ([]string)Atomic: will be replaced during a mergeWhen present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processing of the request. Valid values are: - All: all dry run stages will be processedgracePeriodSeconds (int64)The duration in seconds before the object should be deleted. Value must be non-negative integer. The value zero indicates delete immediately. If this value is nil, the default grace period for the specified type will be used. Defaults to a per object value if not specified. zero means delete immediately.ignoreStoreReadErrorWithClusterBreakingPotential (boolean)if set to true, it will trigger an unsafe deletion of the resource in case the normal deletion flow fails with a corrupt object error. A resource is considered corrupt if it can not be retrieved from the underlying storage successfully because of a) its data can not be transformed e.g. decryption failure, or b) it fails to decode into an object. NOTE: unsafe deletion ignores finalizer constraints, skips precondition checks, and removes the object from the storage. WARNING: This may potentially break the cluster if the workload associated with the resource being unsafe-deleted relies on normal deletion flow. Use only if you REALLY know what you are doing. The default value is false, and the user must opt in to enable itkind (string)Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kindsorphanDependents (boolean)Deprecated: please use the PropagationPolicy, this field will be deprecated in 1.7. Should the dependent objects be orphaned. If true/false, the "orphan" finalizer will be added to/removed from the object's finalizers list. Either this field or PropagationPolicy may be set, but not both.preconditions (Preconditions)Must be fulfilled before a deletion is carried out. If not possible, a 409 Conflict status will be returned.Preconditions must be fulfilled before an operation (update, delete, etc.) is carried out.preconditions.resourceVersion (string)Specifies the target ResourceVersionpreconditions.uid (string)Specifies the target UID.propagationPolicy (string)Whether and how garbage collection will be performed. Either this field or OrphanDependents may be set, but not both. The default policy is decided by the existing finalizer set in the metadata.finalizers and the resource-specific default policy. Acceptable values are: 'Orphan' - orphan the dependents; 'Background' - allow the garbage collector to delete the dependents in the background; 'Foreground' - a cascading policy that deletes all dependents in the foreground.This page is automatically generated.If you plan to report an issue with this page, mention that the page is auto-generated in your issue description. The fix may need to happen elsewhere in the Kubernetes project.


# PKI certificates and requirements

PKI certificates and requirementsKubernetes requires PKI certificates for authentication over TLS. If you install Kubernetes with kubeadm, the certificates that your cluster requires are automatically generated. You can also generate your own certificates -- for example, to keep your private keys more secure by not storing them on the API server. This page explains the certificates that your cluster requires.How certificates are used by your clusterKubernetes requires PKI for the following operations:Server certificatesServer certificate for the API server endpointServer certificate for the etcd serverServer certificates for each kubelet (every node runs a kubelet)Optional server certificate for the front-proxyClient certificatesClient certificates for each kubelet, used to authenticate to the API server as a client of the Kubernetes APIClient certificate for each API server, used to authenticate to etcdClient certificate for the controller manager to securely communicate with the API serverClient certificate for the scheduler to securely communicate with the API serverClient certificates, one for each node, for kube-proxy to authenticate to the API serverOptional client certificates for administrators of the cluster to authenticate to the API serverOptional client certificate for the front-proxyKubelet's server and client certificatesTo establish a secure connection and authenticate itself to the kubelet, the API Server requires a client certificate and key pair.In this scenario, there are two approaches for certificate usage:Shared Certificates: The kube-apiserver can utilize the same certificate and key pair it uses to authenticate its clients. This means that the existing certificates, such as apiserver.crt and apiserver.key, can be used for communicating with the kubelet servers.Separate Certificates: Alternatively, the kube-apiserver can generate a new client certificate and key pair to authenticate its communication with the kubelet servers. In this case, a distinct certificate named kubelet-client.crt and its corresponding private key, kubelet-client.key are created.Note:front-proxy certificates are required only if you run kube-proxy to support an extension API server.etcd also implements mutual TLS to authenticate clients and peers.Where certificates are storedIf you install Kubernetes with kubeadm, most certificates are stored in /etc/kubernetes/pki. All paths in this documentation are relative to that directory, with the exception of user account certificates which kubeadm places in /etc/kubernetes.Configure certificates manuallyIf you don't want kubeadm to generate the required certificates, you can create them using a single root CA or by providing all certificates. See Certificates for details on creating your own certificate authority. See Certificate Management with kubeadm for more on managing certificates.Single root CAYou can create a single root CA, controlled by an administrator. This root CA can then create multiple intermediate CAs, and delegate all further creation to Kubernetes itself.Required CAs:PathDefault CNDescriptionca.crt,keykubernetes-caKubernetes general CAetcd/ca.crt,keyetcd-caFor all etcd-related functionsfront-proxy-ca.crt,keykubernetes-front-proxy-caFor the front-end proxyOn top of the above CAs, it is also necessary to get a public/private key pair for service account management, sa.key and sa.pub. The following example illustrates the CA key and certificate files shown in the previous table:/etc/kubernetes/pki/ca.crt /etc/kubernetes/pki/ca.key /etc/kubernetes/pki/etcd/ca.crt /etc/kubernetes/pki/etcd/ca.key /etc/kubernetes/pki/front-proxy-ca.crt /etc/kubernetes/pki/front-proxy-ca.key All certificatesIf you don't wish to copy the CA private keys to your cluster, you can generate all certificates yourself.Required certificates:Default CNParent CAO (in Subject)kindhosts (SAN)kube-etcdetcd-caserver, client<hostname>, <Host_IP>, localhost, 127.0.0.1kube-etcd-peeretcd-caserver, client<hostname>, <Host_IP>, localhost, 127.0.0.1kube-etcd-healthcheck-clientetcd-caclientkube-apiserver-etcd-clientetcd-caclientkube-apiserverkubernetes-caserver<hostname>, <Host_IP>, <advertise_IP>1kube-apiserver-kubelet-clientkubernetes-casystem:mastersclientfront-proxy-clientkubernetes-front-proxy-caclientNote:Instead of using the super-user group system:masters for kube-apiserver-kubelet-client a less privileged group can be used. kubeadm uses the kubeadm:cluster-admins group for that purpose.where kind maps to one or more of the x509 key usage, which is also documented in the .spec.usages of a CertificateSigningRequest type:kindKey usageserverdigital signature, key encipherment, server authclientdigital signature, key encipherment, client authNote:Hosts/SAN listed above are the recommended ones for getting a working cluster; if required by a specific setup, it is possible to add additional SANs on all the server certificates.Note:For kubeadm users only:The scenario where you are copying to your cluster CA certificates without private keys is referred as external CA in the kubeadm documentation.If you are comparing the above list with a kubeadm generated PKI, please be aware that kube-etcd, kube-etcd-peer and kube-etcd-healthcheck-client certificates are not generated in case of external etcd.Certificate pathsCertificates should be placed in a recommended path (as used by kubeadm). Paths should be specified using the given argument regardless of location.DefaultCNrecommendedkeypathrecommendedcertpathcommandkeyargumentcertargumentetcd-caetcd/ca.keyetcd/ca.crtkube-apiserver--etcd-cafilekube-apiserver-etcd-clientapiserver-etcd-client.keyapiserver-etcd-client.crtkube-apiserver--etcd-keyfile--etcd-certfilekubernetes-caca.keyca.crtkube-apiserver--client-ca-filekubernetes-caca.keyca.crtkube-controller-manager--cluster-signing-key-file--client-ca-file,--root-ca-file,--cluster-signing-cert-filekube-apiserverapiserver.keyapiserver.crtkube-apiserver--tls-private-key-file--tls-cert-filekube-apiserver-kubelet-clientapiserver-kubelet-client.keyapiserver-kubelet-client.crtkube-apiserver--kubelet-client-key--kubelet-client-certificatefront-proxy-cafront-proxy-ca.keyfront-proxy-ca.crtkube-apiserver--requestheader-client-ca-filefront-proxy-cafront-proxy-ca.keyfront-proxy-ca.crtkube-controller-manager--requestheader-client-ca-filefront-proxy-clientfront-proxy-client.keyfront-proxy-client.crtkube-apiserver--proxy-client-key-file--proxy-client-cert-fileetcd-caetcd/ca.keyetcd/ca.crtetcd--trusted-ca-file,--peer-trusted-ca-filekube-etcdetcd/server.keyetcd/server.crtetcd--key-file--cert-filekube-etcd-peeretcd/peer.keyetcd/peer.crtetcd--peer-key-file--peer-cert-fileetcd-caetcd/ca.crtetcdctl--cacertkube-etcd-healthcheck-clientetcd/healthcheck-client.keyetcd/healthcheck-client.crtetcdctl--key--certSame considerations apply for the service account key pair:private key pathpublic key pathcommandargumentsa.keykube-controller-manager--service-account-private-key-filesa.pubkube-apiserver--service-account-key-fileThe following example illustrates the file paths from the previous tables you need to provide if you are generating all of your own keys and certificates:/etc/kubernetes/pki/etcd/ca.key /etc/kubernetes/pki/etcd/ca.crt /etc/kubernetes/pki/apiserver-etcd-client.key /etc/kubernetes/pki/apiserver-etcd-client.crt /etc/kubernetes/pki/ca.key /etc/kubernetes/pki/ca.crt /etc/kubernetes/pki/apiserver.key /etc/kubernetes/pki/apiserver.crt /etc/kubernetes/pki/apiserver-kubelet-client.key /etc/kubernetes/pki/apiserver-kubelet-client.crt /etc/kubernetes/pki/front-proxy-ca.key /etc/kubernetes/pki/front-proxy-ca.crt /etc/kubernetes/pki/front-proxy-client.key /etc/kubernetes/pki/front-proxy-client.crt /etc/kubernetes/pki/etcd/server.key /etc/kubernetes/pki/etcd/server.crt /etc/kubernetes/pki/etcd/peer.key /etc/kubernetes/pki/etcd/peer.crt /etc/kubernetes/pki/etcd/healthcheck-client.key /etc/kubernetes/pki/etcd/healthcheck-client.crt /etc/kubernetes/pki/sa.key /etc/kubernetes/pki/sa.pub Configure certificates for user accountsYou must manually configure these administrator accounts and service accounts:FilenameCredential nameDefault CNO (in Subject)admin.confdefault-adminkubernetes-admin<admin-group>super-admin.confdefault-super-adminkubernetes-super-adminsystem:masterskubelet.confdefault-authsystem:node:<nodeName> (see note)system:nodescontroller-manager.confdefault-controller-managersystem:kube-controller-managerscheduler.confdefault-schedulersystem:kube-schedulerNote:The value of <nodeName> for kubelet.conf must match precisely the value of the node name provided by the kubelet as it registers with the apiserver. For further details, read the Node Authorization.Note:In the above example <admin-group> is implementation specific. Some tools sign the certificate in the default admin.conf to be part of the system:masters group. system:masters is a break-glass, super user group can bypass the authorization layer of Kubernetes, such as RBAC. Also some tools do not generate a separate super-admin.conf with a certificate bound to this super user group.kubeadm generates two separate administrator certificates in kubeconfig files. One is in admin.conf and has Subject: O = kubeadm:cluster-admins, CN = kubernetes-admin. kubeadm:cluster-admins is a custom group bound to the cluster-admin ClusterRole. This file is generated on all kubeadm managed control plane machines.Another is in super-admin.conf that has Subject: O = system:masters, CN = kubernetes-super-admin. This file is generated only on the node where kubeadm init was called.For each configuration, generate an x509 certificate/key pair with the given Common Name (CN) and Organization (O).Run kubectl as follows for each configuration:KUBECONFIG=<filename> kubectl config set-cluster default-cluster --server=https://<host ip>:6443 --certificate-authority <path-to-kubernetes-ca> --embed-certs KUBECONFIG=<filename> kubectl config set-credentials <credential-name> --client-key <path-to-key>.pem --client-certificate <path-to-cert>.pem --embed-certs KUBECONFIG=<filename> kubectl config set-context default-system --cluster default-cluster --user <credential-name> KUBECONFIG=<filename> kubectl config use-context default-system These files are used as follows:FilenameCommandCommentadmin.confkubectlConfigures administrator user for the clustersuper-admin.confkubectlConfigures super administrator user for the clusterkubelet.confkubeletOne required for each node in the cluster.controller-manager.confkube-controller-managerMust be added to manifest in manifests/kube-controller-manager.yamlscheduler.confkube-schedulerMust be added to manifest in manifests/kube-scheduler.yamlThe following files illustrate full paths to the files listed in the previous table:/etc/kubernetes/admin.conf /etc/kubernetes/super-admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf any other IP or DNS name you contact your cluster on (as used by kubeadm the load balancer stable IP and/or DNS name, kubernetes, kubernetes.default, kubernetes.default.svc, kubernetes.default.svc.cluster, kubernetes.default.svc.cluster.local) ↩


# Running Kubelet in Standalone Mode

Running Kubelet in Standalone ModeThis tutorial shows you how to run a standalone kubelet instance.You may have different motivations for running a standalone kubelet. This tutorial is aimed at introducing you to Kubernetes, even if you don't have much experience with it. You can follow this tutorial and learn about node setup, basic (static) Pods, and how Kubernetes manages containers.Once you have followed this tutorial, you could try using a cluster that has a control plane to manage pods and nodes, and other types of objects. For example, Hello, minikube.You can also run the kubelet in standalone mode to suit production use cases, such as to run the control plane for a highly available, resiliently deployed cluster. This tutorial does not cover the details you need for running a resilient control plane.ObjectivesInstall cri-o, and kubelet on a Linux system and run them as systemd services.Launch a Pod running nginx that listens to requests on TCP port 80 on the Pod's IP address.Learn how the different components of the solution interact among themselves.Caution:The kubelet configuration used for this tutorial is insecure by design and should not be used in a production environment.Before you beginAdmin (root) access to a Linux system that uses systemd and iptables (or nftables with iptables emulation).Access to the Internet to download the components needed for the tutorial, such as:A container runtime that implements the Kubernetes (CRI).Network plugins (these are often known as Container Networking Interface (CNI))Required CLI tools: curl, tar, jq.Prepare the systemSwap configurationBy default, kubelet fails to start if swap memory is detected on a node. This means that swap should either be disabled or tolerated by kubelet.Note:If you configure the kubelet to tolerate swap, the kubelet still configures Pods (and the containers in those Pods) not to use swap space. To find out how Pods can actually use the available swap, you can read more about swap memory management on Linux nodes.If you have swap memory enabled, either disable it or add failSwapOn: false to the kubelet configuration file.To check if swap is enabled:sudo swapon --show If there is no output from the command, then swap memory is already disabled.To disable swap temporarily:sudo swapoff -a To make this change persistent across reboots:Make sure swap is disabled in either /etc/fstab or systemd.swap, depending on how it was configured on your system.Enable IPv4 packet forwardingTo check if IPv4 packet forwarding is enabled:cat /proc/sys/net/ipv4/ip_forward If the output is 1, it is already enabled. If the output is 0, then follow next steps.To enable IPv4 packet forwarding, create a configuration file that sets the net.ipv4.ip_forward parameter to 1:sudo tee /etc/sysctl.d/k8s.conf <<EOF net.ipv4.ip_forward = 1 EOF Apply the changes to the system:sudo sysctl --system The output is similar to:... * Applying /etc/sysctl.d/k8s.conf ... net.ipv4.ip_forward = 1 * Applying /etc/sysctl.conf ... Download, install, and configure the componentsNote: This section links to third party projects that provide functionality required by Kubernetes. The Kubernetes project authors aren't responsible for these projects, which are listed alphabetically. To add a project to this list, read the content guide before submitting a change. More information.Install a container runtimeDownload the latest available versions of the required packages (recommended).This tutorial suggests installing the CRI-O container runtime (external link).There are several ways to install the CRI-O container runtime, depending on your particular Linux distribution. Although CRI-O recommends using either deb or rpm packages, this tutorial uses the static binary bundle script of the CRI-O Packaging project, both to streamline the overall process, and to remain distribution agnostic.The script installs and configures additional required software, such as cni-plugins, for container networking, and crun and runc, for running containers.The script will automatically detect your system's processor architecture (amd64 or arm64) and select and install the latest versions of the software packages.Set up CRI-OVisit the releases page (external link).Download the static binary bundle script:curl https://raw.githubusercontent.com/cri-o/packaging/main/get > crio-install Run the installer script:sudo bash crio-install Enable and start the crio service:sudo systemctl daemon-reload sudo systemctl enable --now crio.service Quick test:sudo systemctl is-active crio.service The output is similar to:active Detailed service check:sudo journalctl -f -u crio.service Install network pluginsThe cri-o installer installs and configures the cni-plugins package. You can verify the installation running the following command:/opt/cni/bin/bridge --version The output is similar to:CNI bridge plugin v1.5.1 CNI protocol versions supported: 0.1.0, 0.2.0, 0.3.0, 0.3.1, 0.4.0, 1.0.0 To check the default configuration:cat /etc/cni/net.d/11-crio-ipv4-bridge.conflist The output is similar to:{ "cniVersion": "1.0.0", "name": "crio", "plugins": [ { "type": "bridge", "bridge": "cni0", "isGateway": true, "ipMasq": true, "hairpinMode": true, "ipam": { "type": "host-local", "routes": [ { "dst": "0.0.0.0/0" } ], "ranges": [ [{ "subnet": "10.85.0.0/16" }] ] } } ] } Note:Make sure that the default subnet range (10.85.0.0/16) does not overlap with any of your active networks. If there is an overlap, you can edit the file and change it accordingly. Restart the service after the change.Download and set up the kubeletDownload the latest stable release of the kubelet. curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubelet" curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/arm64/kubelet" Configure:sudo mkdir -p /etc/kubernetes/manifests sudo tee /etc/kubernetes/kubelet.yaml <<EOF apiVersion: kubelet.config.k8s.io/v1beta1 kind: KubeletConfiguration authentication: webhook: enabled: false # Do NOT use in production clusters! authorization: mode: AlwaysAllow # Do NOT use in production clusters! enableServer: false logging: format: text address: 127.0.0.1 # Restrict access to localhost readOnlyPort: 10255 # Do NOT use in production clusters! staticPodPath: /etc/kubernetes/manifests containerRuntimeEndpoint: unix:///var/run/crio/crio.sock EOF Note:Because you are not setting up a production cluster, you are using plain HTTP (readOnlyPort: 10255) for unauthenticated queries to the kubelet's API.The authentication webhook is disabled and authorization mode is set to AlwaysAllow for the purpose of this tutorial. You can learn more about authorization modes and webhook authentication to properly configure kubelet in standalone mode in your environment.See Ports and Protocols to understand which ports Kubernetes components use.Install:chmod +x kubelet sudo cp kubelet /usr/bin/ Create a systemd service unit file:sudo tee /etc/systemd/system/kubelet.service <<EOF [Unit] Description=Kubelet [Service] ExecStart=/usr/bin/kubelet \ --config=/etc/kubernetes/kubelet.yaml Restart=always [Install] WantedBy=multi-user.target EOF The command line argument --kubeconfig has been intentionally omitted in the service configuration file. This argument sets the path to a kubeconfig file that specifies how to connect to the API server, enabling API server mode. Omitting it, enables standalone mode.Enable and start the kubelet service:sudo systemctl daemon-reload sudo systemctl enable --now kubelet.service Quick test:sudo systemctl is-active kubelet.service The output is similar to:active Detailed service check:sudo journalctl -u kubelet.service Check the kubelet's API /healthz endpoint:curl http://localhost:10255/healthz?verbose The output is similar to:[+]ping ok [+]log ok [+]syncloop ok healthz check passed Query the kubelet's API /pods endpoint:curl http://localhost:10255/pods | jq '.' The output is similar to:{ "kind": "PodList", "apiVersion": "v1", "metadata": {}, "items": null } Run a Pod in the kubeletIn standalone mode, you can run Pods using Pod manifests. The manifests can either be on the local filesystem, or fetched via HTTP from a configuration source.Create a manifest for a Pod:cat <<EOF > static-web.yaml apiVersion: v1 kind: Pod metadata: name: static-web spec: containers: - name: web image: nginx ports: - name: web containerPort: 80 protocol: TCP EOF Copy the static-web.yaml manifest file to the /etc/kubernetes/manifests directory.sudo cp static-web.yaml /etc/kubernetes/manifests/ Find out information about the kubelet and the PodThe Pod networking plugin creates a network bridge (cni0) and a pair of veth interfaces for each Pod (one of the pair is inside the newly made Pod, and the other is at the host level).Query the kubelet's API endpoint at http://localhost:10255/pods:curl http://localhost:10255/pods | jq '.' To obtain the IP address of the static-web Pod:curl http://localhost:10255/pods | jq '.items[].status.podIP' The output is similar to:"10.85.0.4" Connect to the nginx server Pod on http://<IP>:<Port> (port 80 is the default), in this case:curl http://10.85.0.4 The output is similar to:<!DOCTYPE html> <html> <head> <title>Welcome to nginx!</title> ... Where to look for more detailsIf you need to diagnose a problem getting this tutorial to work, you can look within the following directories for monitoring and troubleshooting:/var/lib/cni /var/lib/containers /var/lib/kubelet /var/log/containers /var/log/pods Clean upkubeletsudo systemctl disable --now kubelet.service sudo systemctl daemon-reload sudo rm /etc/systemd/system/kubelet.service sudo rm /usr/bin/kubelet sudo rm -rf /etc/kubernetes sudo rm -rf /var/lib/kubelet sudo rm -rf /var/log/containers sudo rm -rf /var/log/pods Container Runtimesudo systemctl disable --now crio.service sudo systemctl daemon-reload sudo rm -rf /usr/local/bin sudo rm -rf /usr/local/lib sudo rm -rf /usr/local/share sudo rm -rf /usr/libexec/crio sudo rm -rf /etc/crio sudo rm -rf /etc/containers Network Pluginssudo rm -rf /opt/cni sudo rm -rf /etc/cni sudo rm -rf /var/lib/cni ConclusionThis page covered the basic aspects of deploying a kubelet in standalone mode. You are now ready to deploy Pods and test additional functionality.Notice that in standalone mode the kubelet does not support fetching Pod configurations from the control plane (because there is no control plane connection).You also cannot use a ConfigMap or a Secret to configure the containers in a static Pod.What's nextFollow Hello, minikube to learn about running Kubernetes with a control plane. The minikube tool helps you set up a practice cluster on your own computer.Learn more about Network PluginsLearn more about Container RuntimesLearn more about kubeletLearn more about static PodsItems on this page refer to third party products or projects that provide functionality required by Kubernetes. The Kubernetes project authors aren't responsible for those third-party products or projects. See the CNCF website guidelines for more details.You should read the content guide before proposing a change that adds an extra third-party link.


# Policy Resources

Policy ResourcesFlowSchemaFlowSchema defines the schema of a group of flows.LimitRangeLimitRange sets resource usage limits for each kind of resource in a Namespace.ResourceQuotaResourceQuota sets aggregate quota restrictions enforced per namespace.NetworkPolicyNetworkPolicy describes what network traffic is allowed for a set of Pods.PodDisruptionBudgetPodDisruptionBudget is an object to define the max disruption that can be caused to a collection of pods.PriorityLevelConfigurationPriorityLevelConfiguration represents the configuration of a priority level.ValidatingAdmissionPolicyValidatingAdmissionPolicy describes the definition of an admission validation policy that accepts or rejects an object without changing it.ValidatingAdmissionPolicyBindingValidatingAdmissionPolicyBinding binds the ValidatingAdmissionPolicy with paramerized resources.MutatingAdmissionPolicy v1beta1MutatingAdmissionPolicy describes the definition of an admission mutation policy that mutates the object coming into admission chain.MutatingAdmissionPolicyBinding v1alpha1MutatingAdmissionPolicyBinding binds the MutatingAdmissionPolicy with parametrized resources.This page is automatically generated.If you plan to report an issue with this page, mention that the page is auto-generated in your issue description. The fix may need to happen elsewhere in the Kubernetes project.


# LeaseCandidate v1beta1

LeaseCandidate v1beta1LeaseCandidate defines a candidate for a Lease object.apiVersion: coordination.k8s.io/v1beta1import "k8s.io/api/coordination/v1beta1"LeaseCandidateLeaseCandidate defines a candidate for a Lease object. Candidates are created such that coordinated leader election will pick the best leader from the list of candidates.apiVersion: coordination.k8s.io/v1beta1kind: LeaseCandidatemetadata (ObjectMeta)More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadataspec (LeaseCandidateSpec)spec contains the specification of the Lease. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-statusLeaseCandidateSpecLeaseCandidateSpec is a specification of a Lease.binaryVersion (string), requiredBinaryVersion is the binary version. It must be in a semver format without leading v. This field is required.leaseName (string), requiredLeaseName is the name of the lease for which this candidate is contending. The limits on this field are the same as on Lease.name. Multiple lease candidates may reference the same Lease.name. This field is immutable.strategy (string), requiredStrategy is the strategy that coordinated leader election will use for picking the leader. If multiple candidates for the same Lease return different strategies, the strategy provided by the candidate with the latest BinaryVersion will be used. If there is still conflict, this is a user error and coordinated leader election will not operate the Lease until resolved.emulationVersion (string)EmulationVersion is the emulation version. It must be in a semver format without leading v. EmulationVersion must be less than or equal to BinaryVersion. This field is required when strategy is "OldestEmulationVersion"pingTime (MicroTime)PingTime is the last time that the server has requested the LeaseCandidate to renew. It is only done during leader election to check if any LeaseCandidates have become ineligible. When PingTime is updated, the LeaseCandidate will respond by updating RenewTime.MicroTime is version of Time with microsecond level precision.renewTime (MicroTime)RenewTime is the time that the LeaseCandidate was last updated. Any time a Lease needs to do leader election, the PingTime field is updated to signal to the LeaseCandidate that they should update the RenewTime. Old LeaseCandidate objects are also garbage collected if it has been hours since the last renew. The PingTime field is updated regularly to prevent garbage collection for still active LeaseCandidates.MicroTime is version of Time with microsecond level precision.LeaseCandidateListLeaseCandidateList is a list of Lease objects.apiVersion: coordination.k8s.io/v1beta1kind: LeaseCandidateListmetadata (ListMeta)Standard list metadata. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadataitems ([]LeaseCandidate), requireditems is a list of schema objects.Operationsget read the specified LeaseCandidateHTTP RequestGET /apis/coordination.k8s.io/v1beta1/namespaces/{namespace}/leasecandidates/{name}Parametersname (in path): string, requiredname of the LeaseCandidatenamespace (in path): string, requirednamespacepretty (in query): stringprettyResponse200 (LeaseCandidate): OK401: Unauthorizedlist list or watch objects of kind LeaseCandidateHTTP RequestGET /apis/coordination.k8s.io/v1beta1/namespaces/{namespace}/leasecandidatesParametersnamespace (in path): string, requirednamespaceallowWatchBookmarks (in query): booleanallowWatchBookmarkscontinue (in query): stringcontinuefieldSelector (in query): stringfieldSelectorlabelSelector (in query): stringlabelSelectorlimit (in query): integerlimitpretty (in query): stringprettyresourceVersion (in query): stringresourceVersionresourceVersionMatch (in query): stringresourceVersionMatchsendInitialEvents (in query): booleansendInitialEventstimeoutSeconds (in query): integertimeoutSecondswatch (in query): booleanwatchResponse200 (LeaseCandidateList): OK401: Unauthorizedlist list or watch objects of kind LeaseCandidateHTTP RequestGET /apis/coordination.k8s.io/v1beta1/leasecandidatesParametersallowWatchBookmarks (in query): booleanallowWatchBookmarkscontinue (in query): stringcontinuefieldSelector (in query): stringfieldSelectorlabelSelector (in query): stringlabelSelectorlimit (in query): integerlimitpretty (in query): stringprettyresourceVersion (in query): stringresourceVersionresourceVersionMatch (in query): stringresourceVersionMatchsendInitialEvents (in query): booleansendInitialEventstimeoutSeconds (in query): integertimeoutSecondswatch (in query): booleanwatchResponse200 (LeaseCandidateList): OK401: Unauthorizedcreate create a LeaseCandidateHTTP RequestPOST /apis/coordination.k8s.io/v1beta1/namespaces/{namespace}/leasecandidatesParametersnamespace (in path): string, requirednamespacebody: LeaseCandidate, requireddryRun (in query): stringdryRunfieldManager (in query): stringfieldManagerfieldValidation (in query): stringfieldValidationpretty (in query): stringprettyResponse200 (LeaseCandidate): OK201 (LeaseCandidate): Created202 (LeaseCandidate): Accepted401: Unauthorizedupdate replace the specified LeaseCandidateHTTP RequestPUT /apis/coordination.k8s.io/v1beta1/namespaces/{namespace}/leasecandidates/{name}Parametersname (in path): string, requiredname of the LeaseCandidatenamespace (in path): string, requirednamespacebody: LeaseCandidate, requireddryRun (in query): stringdryRunfieldManager (in query): stringfieldManagerfieldValidation (in query): stringfieldValidationpretty (in query): stringprettyResponse200 (LeaseCandidate): OK201 (LeaseCandidate): Created401: Unauthorizedpatch partially update the specified LeaseCandidateHTTP RequestPATCH /apis/coordination.k8s.io/v1beta1/namespaces/{namespace}/leasecandidates/{name}Parametersname (in path): string, requiredname of the LeaseCandidatenamespace (in path): string, requirednamespacebody: Patch, requireddryRun (in query): stringdryRunfieldManager (in query): stringfieldManagerfieldValidation (in query): stringfieldValidationforce (in query): booleanforcepretty (in query): stringprettyResponse200 (LeaseCandidate): OK201 (LeaseCandidate): Created401: Unauthorizeddelete delete a LeaseCandidateHTTP RequestDELETE /apis/coordination.k8s.io/v1beta1/namespaces/{namespace}/leasecandidates/{name}Parametersname (in path): string, requiredname of the LeaseCandidatenamespace (in path): string, requirednamespacebody: DeleteOptionsdryRun (in query): stringdryRungracePeriodSeconds (in query): integergracePeriodSecondsignoreStoreReadErrorWithClusterBreakingPotential (in query): booleanignoreStoreReadErrorWithClusterBreakingPotentialpretty (in query): stringprettypropagationPolicy (in query): stringpropagationPolicyResponse200 (Status): OK202 (Status): Accepted401: Unauthorizeddeletecollection delete collection of LeaseCandidateHTTP RequestDELETE /apis/coordination.k8s.io/v1beta1/namespaces/{namespace}/leasecandidatesParametersnamespace (in path): string, requirednamespacebody: DeleteOptionscontinue (in query): stringcontinuedryRun (in query): stringdryRunfieldSelector (in query): stringfieldSelectorgracePeriodSeconds (in query): integergracePeriodSecondsignoreStoreReadErrorWithClusterBreakingPotential (in query): booleanignoreStoreReadErrorWithClusterBreakingPotentiallabelSelector (in query): stringlabelSelectorlimit (in query): integerlimitpretty (in query): stringprettypropagationPolicy (in query): stringpropagationPolicyresourceVersion (in query): stringresourceVersionresourceVersionMatch (in query): stringresourceVersionMatchsendInitialEvents (in query): booleansendInitialEventstimeoutSeconds (in query): integertimeoutSecondsResponse200 (Status): OK401: UnauthorizedThis page is automatically generated.If you plan to report an issue with this page, mention that the page is auto-generated in your issue description. The fix may need to happen elsewhere in the Kubernetes project.


# Contributing to the Upstream Kubernetes Code

Contributing to the Upstream Kubernetes CodeThis page shows how to contribute to the upstream kubernetes/kubernetes project. You can fix bugs found in the Kubernetes API documentation or the content of the Kubernetes components such as kubeadm, kube-apiserver, and kube-controller-manager.If you instead want to regenerate the reference documentation for the Kubernetes API or the kube-* components from the upstream code, see the following instructions:Generating Reference Documentation for the Kubernetes APIGenerating Reference Documentation for the Kubernetes Components and ToolsBefore you beginYou need to have these tools installed:GitGolang version 1.13+Dockeretcdmakegcc compiler/linkerYour GOPATH environment variable must be set, and the location of etcd must be in your PATH environment variable.You need to know how to create a pull request to a GitHub repository. Typically, this involves creating a fork of the repository. For more information, see Creating a Pull Request and GitHub Standard Fork & Pull Request Workflow.The big pictureThe reference documentation for the Kubernetes API and the kube-* components such as kube-apiserver, kube-controller-manager are automatically generated from the source code in the upstream Kubernetes.When you see bugs in the generated documentation, you may want to consider creating a patch to fix it in the upstream project.Clone the Kubernetes repositoryIf you don't already have the kubernetes/kubernetes repository, get it now:mkdir $GOPATH/src cd $GOPATH/src go get github.com/kubernetes/kubernetes Determine the base directory of your clone of the kubernetes/kubernetes repository. For example, if you followed the preceding step to get the repository, your base directory is $GOPATH/src/github.com/kubernetes/kubernetes. The remaining steps refer to your base directory as <k8s-base>.Determine the base directory of your clone of the kubernetes-sigs/reference-docs repository. For example, if you followed the preceding step to get the repository, your base directory is $GOPATH/src/github.com/kubernetes-sigs/reference-docs. The remaining steps refer to your base directory as <rdocs-base>.Edit the Kubernetes source codeThe Kubernetes API reference documentation is automatically generated from an OpenAPI spec, which is generated from the Kubernetes source code. If you want to change the API reference documentation, the first step is to change one or more comments in the Kubernetes source code.The documentation for the kube-* components is also generated from the upstream source code. You must change the code related to the component you want to fix in order to fix the generated documentation.Make changes to the upstream source codeNote:The following steps are an example, not a general procedure. Details will be different in your situation.Here's an example of editing a comment in the Kubernetes source code.In your local kubernetes/kubernetes repository, check out the default branch, and make sure it is up to date:cd <k8s-base> git checkout master git pull https://github.com/kubernetes/kubernetes master Suppose this source file in that default branch has the typo "atmost":kubernetes/kubernetes/staging/src/k8s.io/api/apps/v1/types.goIn your local environment, open types.go, and change "atmost" to "at most".Verify that you have changed the file:git status The output shows that you are on the master branch, and that the types.go source file has been modified:On branch master ... modified: staging/src/k8s.io/api/apps/v1/types.go Commit your edited fileRun git add and git commit to commit the changes you have made so far. In the next step, you will do a second commit. It is important to keep your changes separated into two commits.Generate the OpenAPI spec and related filesGo to <k8s-base> and run these scripts:./hack/update-codegen.sh ./hack/update-openapi-spec.sh Run git status to see what was generated.On branch master ... modified: api/openapi-spec/swagger.json modified: api/openapi-spec/v3/apis__apps__v1_openapi.json modified: pkg/generated/openapi/zz_generated.openapi.go modified: staging/src/k8s.io/api/apps/v1/generated.proto modified: staging/src/k8s.io/api/apps/v1/types_swagger_doc_generated.go View the contents of api/openapi-spec/swagger.json to make sure the typo is fixed. For example, you could run git diff -a api/openapi-spec/swagger.json. This is important, because swagger.json is the input to the second stage of the doc generation process.Run git add and git commit to commit your changes. Now you have two commits: one that contains the edited types.go file, and one that contains the generated OpenAPI spec and related files. Keep these two commits separate. That is, do not squash your commits.Submit your changes as a pull request to the master branch of the kubernetes/kubernetes repository. Monitor your pull request, and respond to reviewer comments as needed. Continue to monitor your pull request until it is merged.PR 57758 is an example of a pull request that fixes a typo in the Kubernetes source code.Note:It can be tricky to determine the correct source file to be changed. In the preceding example, the authoritative source file is in the staging directory in the kubernetes/kubernetes repository. But in your situation,the staging directory might not be the place to find the authoritative source. For guidance, check the README files in kubernetes/kubernetes repository and in related repositories, such as kubernetes/apiserver.Cherry pick your commit into a release branchIn the preceding section, you edited a file in the master branch and then ran scripts to generate an OpenAPI spec and related files. Then you submitted your changes in a pull request to the master branch of the kubernetes/kubernetes repository. Now suppose you want to backport your change into a release branch. For example, suppose the master branch is being used to develop Kubernetes version 1.34, and you want to backport your change into the release-1.33 branch.Recall that your pull request has two commits: one for editing types.go and one for the files generated by scripts. The next step is to propose a cherry pick of your first commit into the release-1.33 branch. The idea is to cherry pick the commit that edited types.go, but not the commit that has the results of running the scripts. For instructions, see Propose a Cherry Pick.Note:Proposing a cherry pick requires that you have permission to set a label and a milestone in your pull request. If you don't have those permissions, you will need to work with someone who can set the label and milestone for you.When you have a pull request in place for cherry picking your one commit into the release-1.33 branch, the next step is to run these scripts in the release-1.33 branch of your local environment../hack/update-codegen.sh ./hack/update-openapi-spec.sh Now add a commit to your cherry-pick pull request that has the recently generated OpenAPI spec and related files. Monitor your pull request until it gets merged into the release-1.33 branch.At this point, both the master branch and the release-1.33 branch have your updated types.go file and a set of generated files that reflect the change you made to types.go. Note that the generated OpenAPI spec and other generated files in the release-1.33 branch are not necessarily the same as the generated files in the master branch. The generated files in the release-1.33 branch contain API elements only from Kubernetes 1.33. The generated files in the master branch might contain API elements that are not in 1.33, but are under development for 1.34.Generate the published reference docsThe preceding section showed how to edit a source file and then generate several files, including api/openapi-spec/swagger.json in the kubernetes/kubernetes repository. The swagger.json file is the OpenAPI definition file to use for generating the API reference documentation.You are now ready to follow the Generating Reference Documentation for the Kubernetes API guide to generate the published Kubernetes API reference documentation.What's nextGenerating Reference Documentation for the Kubernetes APIGenerating Reference Docs for Kubernetes Components and ToolsGenerating Reference Documentation for kubectl Commands