JobJob represents the configuration of a single job.apiVersion: batch/v1import "k8s.io/api/batch/v1"JobJob represents the configuration of a single job.apiVersion: batch/v1kind: Jobmetadata (ObjectMeta)Standard object's metadata. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadataspec (JobSpec)Specification of the desired behavior of a job. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-statusstatus (JobStatus)Current status of a job. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-statusJobSpecJobSpec describes how the job execution will look like.Replicastemplate (PodTemplateSpec), requiredDescribes the pod that will be created when executing a job. The only allowed template.spec.restartPolicy values are "Never" or "OnFailure". More info: https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/parallelism (int32)Specifies the maximum desired number of pods the job should run at any given time. The actual number of pods running in steady state will be less than this number when ((.spec.completions - .status.successful) < .spec.parallelism), i.e. when the work left to do is less than max parallelism. More info: https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/Lifecyclecompletions (int32)Specifies the desired number of successfully finished pods the job should be run with. Setting to null means that the success of any pod signals the success of all pods, and allows parallelism to have any positive value. Setting to 1 means that parallelism is limited to 1 and the success of that pod signals the success of the job. More info: https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/completionMode (string)completionMode specifies how Pod completions are tracked. It can be NonIndexed (default) or Indexed.NonIndexed means that the Job is considered complete when there have been .spec.completions successfully completed Pods. Each Pod completion is homologous to each other.Indexed means that the Pods of a Job get an associated completion index from 0 to (.spec.completions - 1), available in the annotation batch.kubernetes.io/job-completion-index. The Job is considered complete when there is one successfully completed Pod for each index. When value is Indexed, .spec.completions must be specified and .spec.parallelism must be less than or equal to 10^5. In addition, The Pod name takes the form $(job-name)-$(index)-$(random-string), the Pod hostname takes the form $(job-name)-$(index).More completion modes can be added in the future. If the Job controller observes a mode that it doesn't recognize, which is possible during upgrades due to version skew, the controller skips updates for the Job.backoffLimit (int32)Specifies the number of retries before marking this job failed. Defaults to 6, unless backoffLimitPerIndex (only Indexed Job) is specified. When backoffLimitPerIndex is specified, backoffLimit defaults to 2147483647.activeDeadlineSeconds (int64)Specifies the duration in seconds relative to the startTime that the job may be continuously active before the system tries to terminate it; value must be positive integer. If a Job is suspended (at creation or through an update), this timer will effectively be stopped and reset when the Job is resumed again.ttlSecondsAfterFinished (int32)ttlSecondsAfterFinished limits the lifetime of a Job that has finished execution (either Complete or Failed). If this field is set, ttlSecondsAfterFinished after the Job finishes, it is eligible to be automatically deleted. When the Job is being deleted, its lifecycle guarantees (e.g. finalizers) will be honored. If this field is unset, the Job won't be automatically deleted. If this field is set to zero, the Job becomes eligible to be deleted immediately after it finishes.suspend (boolean)suspend specifies whether the Job controller should create Pods or not. If a Job is created with suspend set to true, no Pods are created by the Job controller. If a Job is suspended after creation (i.e. the flag goes from false to true), the Job controller will delete all active Pods associated with this Job. Users must design their workload to gracefully handle this. Suspending a Job will reset the StartTime field of the Job, effectively resetting the ActiveDeadlineSeconds timer too. Defaults to false.Selectorselector (LabelSelector)A label query over pods that should match the pod count. Normally, the system sets this field for you. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#label-selectorsmanualSelector (boolean)manualSelector controls generation of pod labels and pod selectors. Leave manualSelector unset unless you are certain what you are doing. When false or unset, the system pick labels unique to this job and appends those labels to the pod template. When true, the user is responsible for picking unique labels and specifying the selector. Failure to pick a unique label may cause this and other jobs to not function correctly. However, You may see manualSelector=true in jobs that were created with the old extensions/v1beta1 API. More info: https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/#specifying-your-own-pod-selectorBeta levelpodFailurePolicy (PodFailurePolicy)Specifies the policy of handling failed pods. In particular, it allows to specify the set of actions and conditions which need to be satisfied to take the associated action. If empty, the default behaviour applies - the counter of failed pods, represented by the jobs's .status.failed field, is incremented and it is checked against the backoffLimit. This field cannot be used in combination with restartPolicy=OnFailure.PodFailurePolicy describes how failed pods influence the backoffLimit.podFailurePolicy.rules ([]PodFailurePolicyRule), requiredAtomic: will be replaced during a mergeA list of pod failure policy rules. The rules are evaluated in order. Once a rule matches a Pod failure, the remaining of the rules are ignored. When no rule matches the Pod failure, the default handling applies - the counter of pod failures is incremented and it is checked against the backoffLimit. At most 20 elements are allowed.PodFailurePolicyRule describes how a pod failure is handled when the requirements are met. One of onExitCodes and onPodConditions, but not both, can be used in each rule.podFailurePolicy.rules.action (string), requiredSpecifies the action taken on a pod failure when the requirements are satisfied. Possible values are:FailJob: indicates that the pod's job is marked as Failed and all running pods are terminated.FailIndex: indicates that the pod's index is marked as Failed and will not be restarted.Ignore: indicates that the counter towards the .backoffLimit is not incremented and a replacement pod is created.Count: indicates that the pod is handled in the default way - the counter towards the .backoffLimit is incremented. Additional values are considered to be added in the future. Clients should react to an unknown action by skipping the rule.podFailurePolicy.rules.onExitCodes (PodFailurePolicyOnExitCodesRequirement)Represents the requirement on the container exit codes.PodFailurePolicyOnExitCodesRequirement describes the requirement for handling a failed pod based on its container exit codes. In particular, it lookups the .state.terminated.exitCode for each app container and init container status, represented by the .status.containerStatuses and .status.initContainerStatuses fields in the Pod status, respectively. Containers completed with success (exit code 0) are excluded from the requirement check.podFailurePolicy.rules.onExitCodes.operator (string), requiredRepresents the relationship between the container exit code(s) and the specified values. Containers completed with success (exit code 0) are excluded from the requirement check. Possible values are:In: the requirement is satisfied if at least one container exit code (might be multiple if there are multiple containers not restricted by the 'containerName' field) is in the set of specified values.NotIn: the requirement is satisfied if at least one container exit code (might be multiple if there are multiple containers not restricted by the 'containerName' field) is not in the set of specified values. Additional values are considered to be added in the future. Clients should react to an unknown operator by assuming the requirement is not satisfied.podFailurePolicy.rules.onExitCodes.values ([]int32), requiredSet: unique values will be kept during a mergeSpecifies the set of values. Each returned container exit code (might be multiple in case of multiple containers) is checked against this set of values with respect to the operator. The list of values must be ordered and must not contain duplicates. Value '0' cannot be used for the In operator. At least one element is required. At most 255 elements are allowed.podFailurePolicy.rules.onExitCodes.containerName (string)Restricts the check for exit codes to the container with the specified name. When null, the rule applies to all containers. When specified, it should match one the container or initContainer names in the pod template.podFailurePolicy.rules.onPodConditions ([]PodFailurePolicyOnPodConditionsPattern)Atomic: will be replaced during a mergeRepresents the requirement on the pod conditions. The requirement is represented as a list of pod condition patterns. The requirement is satisfied if at least one pattern matches an actual pod condition. At most 20 elements are allowed.PodFailurePolicyOnPodConditionsPattern describes a pattern for matching an actual pod condition type.podFailurePolicy.rules.onPodConditions.status (string), requiredSpecifies the required Pod condition status. To match a pod condition it is required that the specified status equals the pod condition status. Defaults to True.podFailurePolicy.rules.onPodConditions.type (string), requiredSpecifies the required Pod condition type. To match a pod condition it is required that specified type equals the pod condition type.successPolicy (SuccessPolicy)successPolicy specifies the policy when the Job can be declared as succeeded. If empty, the default behavior applies - the Job is declared as succeeded only when the number of succeeded pods equals to the completions. When the field is specified, it must be immutable and works only for the Indexed Jobs. Once the Job meets the SuccessPolicy, the lingering pods are terminated.SuccessPolicy describes when a Job can be declared as succeeded based on the success of some indexes.successPolicy.rules ([]SuccessPolicyRule), requiredAtomic: will be replaced during a mergerules represents the list of alternative rules for the declaring the Jobs as successful before .status.succeeded >= .spec.completions. Once any of the rules are met, the "SuccessCriteriaMet" condition is added, and the lingering pods are removed. The terminal state for such a Job has the "Complete" condition. Additionally, these rules are evaluated in order; Once the Job meets one of the rules, other rules are ignored. At most 20 elements are allowed.SuccessPolicyRule describes rule for declaring a Job as succeeded. Each rule must have at least one of the "succeededIndexes" or "succeededCount" specified.successPolicy.rules.succeededCount (int32)succeededCount specifies the minimal required size of the actual set of the succeeded indexes for the Job. When succeededCount is used along with succeededIndexes, the check is constrained only to the set of indexes specified by succeededIndexes. For example, given that succeededIndexes is "1-4", succeededCount is "3", and completed indexes are "1", "3", and "5", the Job isn't declared as succeeded because only "1" and "3" indexes are considered in that rules. When this field is null, this doesn't default to any value and is never evaluated at any time. When specified it needs to be a positive integer.successPolicy.rules.succeededIndexes (string)succeededIndexes specifies the set of indexes which need to be contained in the actual set of the succeeded indexes for the Job. The list of indexes must be within 0 to ".spec.completions-1" and must not contain duplicates. At least one element is required. The indexes are represented as intervals separated by commas. The intervals can be a decimal integer or a pair of decimal integers separated by a hyphen. The number are listed in represented by the first and last element of the series, separated by a hyphen. For example, if the completed indexes are 1, 3, 4, 5 and 7, they are represented as "1,3-5,7". When this field is null, this field doesn't default to any value and is never evaluated at any time.Alpha levelbackoffLimitPerIndex (int32)Specifies the limit for the number of retries within an index before marking this index as failed. When enabled the number of failures per index is kept in the pod's batch.kubernetes.io/job-index-failure-count annotation. It can only be set when Job's completionMode=Indexed, and the Pod's restart policy is Never. The field is immutable.managedBy (string)ManagedBy field indicates the controller that manages a Job. The k8s Job controller reconciles jobs which don't have this field at all or the field value is the reserved string kubernetes.io/job-controller, but skips reconciling Jobs with a custom value for this field. The value must be a valid domain-prefixed path (e.g. acme.io/foo) - all characters before the first "/" must be a valid subdomain as defined by RFC 1123. All characters trailing the first "/" must be valid HTTP Path characters as defined by RFC 3986. The value cannot exceed 63 characters. This field is immutable.This field is beta-level. The job controller accepts setting the field when the feature gate JobManagedBy is enabled (enabled by default).maxFailedIndexes (int32)Specifies the maximal number of failed indexes before marking the Job as failed, when backoffLimitPerIndex is set. Once the number of failed indexes exceeds this number the entire Job is marked as Failed and its execution is terminated. When left as null the job continues execution of all of its indexes and is marked with the Complete Job condition. It can only be specified when backoffLimitPerIndex is set. It can be null or up to completions. It is required and must be less than or equal to 10^4 when is completions greater than 10^5.podReplacementPolicy (string)podReplacementPolicy specifies when to create replacement Pods. Possible values are: - TerminatingOrFailed means that we recreate pods when they are terminating (has a metadata.deletionTimestamp) or failed.Failed means to wait until a previously created Pod is fully terminated (has phase Failed or Succeeded) before creating a replacement Pod.When using podFailurePolicy, Failed is the the only allowed value. TerminatingOrFailed and Failed are allowed values when podFailurePolicy is not in use.JobStatusJobStatus represents the current state of a Job.startTime (Time)Represents time when the job controller started processing a job. When a Job is created in the suspended state, this field is not set until the first time it is resumed. This field is reset every time a Job is resumed from suspension. It is represented in RFC3339 form and is in UTC.Once set, the field can only be removed when the job is suspended. The field cannot be modified while the job is unsuspended or finished.Time is a wrapper around time.Time which supports correct marshaling to YAML and JSON. Wrappers are provided for many of the factory methods that the time package offers.completionTime (Time)Represents time when the job was completed. It is not guaranteed to be set in happens-before order across separate operations. It is represented in RFC3339 form and is in UTC. The completion time is set when the job finishes successfully, and only then. The value cannot be updated or removed. The value indicates the same or later point in time as the startTime field.Time is a wrapper around time.Time which supports correct marshaling to YAML and JSON. Wrappers are provided for many of the factory methods that the time package offers.active (int32)The number of pending and running pods which are not terminating (without a deletionTimestamp). The value is zero for finished jobs.failed (int32)The number of pods which reached phase Failed. The value increases monotonically.succeeded (int32)The number of pods which reached phase Succeeded. The value increases monotonically for a given spec. However, it may decrease in reaction to scale down of elastic indexed jobs.completedIndexes (string)completedIndexes holds the completed indexes when .spec.completionMode = "Indexed" in a text format. The indexes are represented as decimal integers separated by commas. The numbers are listed in increasing order. Three or more consecutive numbers are compressed and represented by the first and last element of the series, separated by a hyphen. For example, if the completed indexes are 1, 3, 4, 5 and 7, they are represented as "1,3-5,7".conditions ([]JobCondition)Patch strategy: merge on key typeAtomic: will be replaced during a mergeThe latest available observations of an object's current state. When a Job fails, one of the conditions will have type "Failed" and status true. When a Job is suspended, one of the conditions will have type "Suspended" and status true; when the Job is resumed, the status of this condition will become false. When a Job is completed, one of the conditions will have type "Complete" and status true.A job is considered finished when it is in a terminal condition, either "Complete" or "Failed". A Job cannot have both the "Complete" and "Failed" conditions. Additionally, it cannot be in the "Complete" and "FailureTarget" conditions. The "Complete", "Failed" and "FailureTarget" conditions cannot be disabled.More info: https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/JobCondition describes current state of a job.conditions.status (string), requiredStatus of the condition, one of True, False, Unknown.conditions.type (string), requiredType of job condition, Complete or Failed.conditions.lastProbeTime (Time)Last time the condition was checked.Time is a wrapper around time.Time which supports correct marshaling to YAML and JSON. Wrappers are provided for many of the factory methods that the time package offers.conditions.lastTransitionTime (Time)Last time the condition transit from one status to another.Time is a wrapper around time.Time which supports correct marshaling to YAML and JSON. Wrappers are provided for many of the factory methods that the time package offers.conditions.message (string)Human readable message indicating details about last transition.conditions.reason (string)(brief) reason for the condition's last transition.uncountedTerminatedPods (UncountedTerminatedPods)uncountedTerminatedPods holds the UIDs of Pods that have terminated but the job controller hasn't yet accounted for in the status counters.The job controller creates pods with a finalizer. When a pod terminates (succeeded or failed), the controller does three steps to account for it in the job status:Add the pod UID to the arrays in this field. 2. Remove the pod finalizer. 3. Remove the pod UID from the arrays while increasing the corresponding counter.Old jobs might not be tracked using this field, in which case the field remains null. The structure is empty for finished jobs.UncountedTerminatedPods holds UIDs of Pods that have terminated but haven't been accounted in Job status counters.uncountedTerminatedPods.failed ([]string)Set: unique values will be kept during a mergefailed holds UIDs of failed Pods.uncountedTerminatedPods.succeeded ([]string)Set: unique values will be kept during a mergesucceeded holds UIDs of succeeded Pods.Beta levelready (int32)The number of active pods which have a Ready condition and are not terminating (without a deletionTimestamp).Alpha levelfailedIndexes (string)FailedIndexes holds the failed indexes when spec.backoffLimitPerIndex is set. The indexes are represented in the text format analogous as for the completedIndexes field, ie. they are kept as decimal integers separated by commas. The numbers are listed in increasing order. Three or more consecutive numbers are compressed and represented by the first and last element of the series, separated by a hyphen. For example, if the failed indexes are 1, 3, 4, 5 and 7, they are represented as "1,3-5,7". The set of failed indexes cannot overlap with the set of completed indexes.terminating (int32)The number of pods which are terminating (in phase Pending or Running and have a deletionTimestamp).This field is beta-level. The job controller populates the field when the feature gate JobPodReplacementPolicy is enabled (enabled by default).JobListJobList is a collection of jobs.apiVersion: batch/v1kind: JobListmetadata (ListMeta)Standard list metadata. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadataitems ([]Job), requireditems is the list of Jobs.Operationsget read the specified JobHTTP RequestGET /apis/batch/v1/namespaces/{namespace}/jobs/{name}Parametersname (in path): string, requiredname of the Jobnamespace (in path): string, requirednamespacepretty (in query): stringprettyResponse200 (Job): OK401: Unauthorizedget read status of the specified JobHTTP RequestGET /apis/batch/v1/namespaces/{namespace}/jobs/{name}/statusParametersname (in path): string, requiredname of the Jobnamespace (in path): string, requirednamespacepretty (in query): stringprettyResponse200 (Job): OK401: Unauthorizedlist list or watch objects of kind JobHTTP RequestGET /apis/batch/v1/namespaces/{namespace}/jobsParametersnamespace (in path): string, requirednamespaceallowWatchBookmarks (in query): booleanallowWatchBookmarkscontinue (in query): stringcontinuefieldSelector (in query): stringfieldSelectorlabelSelector (in query): stringlabelSelectorlimit (in query): integerlimitpretty (in query): stringprettyresourceVersion (in query): stringresourceVersionresourceVersionMatch (in query): stringresourceVersionMatchsendInitialEvents (in query): booleansendInitialEventstimeoutSeconds (in query): integertimeoutSecondswatch (in query): booleanwatchResponse200 (JobList): OK401: Unauthorizedlist list or watch objects of kind JobHTTP RequestGET /apis/batch/v1/jobsParametersallowWatchBookmarks (in query): booleanallowWatchBookmarkscontinue (in query): stringcontinuefieldSelector (in query): stringfieldSelectorlabelSelector (in query): stringlabelSelectorlimit (in query): integerlimitpretty (in query): stringprettyresourceVersion (in query): stringresourceVersionresourceVersionMatch (in query): stringresourceVersionMatchsendInitialEvents (in query): booleansendInitialEventstimeoutSeconds (in query): integertimeoutSecondswatch (in query): booleanwatchResponse200 (JobList): OK401: Unauthorizedcreate create a JobHTTP RequestPOST /apis/batch/v1/namespaces/{namespace}/jobsParametersnamespace (in path): string, requirednamespacebody: Job, requireddryRun (in query): stringdryRunfieldManager (in query): stringfieldManagerfieldValidation (in query): stringfieldValidationpretty (in query): stringprettyResponse200 (Job): OK201 (Job): Created202 (Job): Accepted401: Unauthorizedupdate replace the specified JobHTTP RequestPUT /apis/batch/v1/namespaces/{namespace}/jobs/{name}Parametersname (in path): string, requiredname of the Jobnamespace (in path): string, requirednamespacebody: Job, requireddryRun (in query): stringdryRunfieldManager (in query): stringfieldManagerfieldValidation (in query): stringfieldValidationpretty (in query): stringprettyResponse200 (Job): OK201 (Job): Created401: Unauthorizedupdate replace status of the specified JobHTTP RequestPUT /apis/batch/v1/namespaces/{namespace}/jobs/{name}/statusParametersname (in path): string, requiredname of the Jobnamespace (in path): string, requirednamespacebody: Job, requireddryRun (in query): stringdryRunfieldManager (in query): stringfieldManagerfieldValidation (in query): stringfieldValidationpretty (in query): stringprettyResponse200 (Job): OK201 (Job): Created401: Unauthorizedpatch partially update the specified JobHTTP RequestPATCH /apis/batch/v1/namespaces/{namespace}/jobs/{name}Parametersname (in path): string, requiredname of the Jobnamespace (in path): string, requirednamespacebody: Patch, requireddryRun (in query): stringdryRunfieldManager (in query): stringfieldManagerfieldValidation (in query): stringfieldValidationforce (in query): booleanforcepretty (in query): stringprettyResponse200 (Job): OK201 (Job): Created401: Unauthorizedpatch partially update status of the specified JobHTTP RequestPATCH /apis/batch/v1/namespaces/{namespace}/jobs/{name}/statusParametersname (in path): string, requiredname of the Jobnamespace (in path): string, requirednamespacebody: Patch, requireddryRun (in query): stringdryRunfieldManager (in query): stringfieldManagerfieldValidation (in query): stringfieldValidationforce (in query): booleanforcepretty (in query): stringprettyResponse200 (Job): OK201 (Job): Created401: Unauthorizeddelete delete a JobHTTP RequestDELETE /apis/batch/v1/namespaces/{namespace}/jobs/{name}Parametersname (in path): string, requiredname of the Jobnamespace (in path): string, requirednamespacebody: DeleteOptionsdryRun (in query): stringdryRungracePeriodSeconds (in query): integergracePeriodSecondsignoreStoreReadErrorWithClusterBreakingPotential (in query): booleanignoreStoreReadErrorWithClusterBreakingPotentialpretty (in query): stringprettypropagationPolicy (in query): stringpropagationPolicyResponse200 (Status): OK202 (Status): Accepted401: Unauthorizeddeletecollection delete collection of JobHTTP RequestDELETE /apis/batch/v1/namespaces/{namespace}/jobsParametersnamespace (in path): string, requirednamespacebody: DeleteOptionscontinue (in query): stringcontinuedryRun (in query): stringdryRunfieldSelector (in query): stringfieldSelectorgracePeriodSeconds (in query): integergracePeriodSecondsignoreStoreReadErrorWithClusterBreakingPotential (in query): booleanignoreStoreReadErrorWithClusterBreakingPotentiallabelSelector (in query): stringlabelSelectorlimit (in query): integerlimitpretty (in query): stringprettypropagationPolicy (in query): stringpropagationPolicyresourceVersion (in query): stringresourceVersionresourceVersionMatch (in query): stringresourceVersionMatchsendInitialEvents (in query): booleansendInitialEventstimeoutSeconds (in query): integertimeoutSecondsResponse200 (Status): OK401: UnauthorizedThis page is automatically generated.If you plan to report an issue with this page, mention that the page is auto-generated in your issue description. The fix may need to happen elsewhere in the Kubernetes project.

Job


FinalizersFinalizers are namespaced keys that tell Kubernetes to wait until specific conditions are met before it fully deletes resources that are marked for deletion. Finalizers alert controllers to clean up resources the deleted object owned.When you tell Kubernetes to delete an object that has finalizers specified for it, the Kubernetes API marks the object for deletion by populating .metadata.deletionTimestamp, and returns a 202 status code (HTTP "Accepted"). The target object remains in a terminating state while the control plane, or other components, take the actions defined by the finalizers. After these actions are complete, the controller removes the relevant finalizers from the target object. When the metadata.finalizers field is empty, Kubernetes considers the deletion complete and deletes the object.You can use finalizers to control garbage collection of resources. For example, you can define a finalizer to clean up related API resources or infrastructure before the controller deletes the object being finalized.You can use finalizers to control garbage collection of objects by alerting controllers to perform specific cleanup tasks before deleting the target resource.Finalizers don't usually specify the code to execute. Instead, they are typically lists of keys on a specific resource similar to annotations. Kubernetes specifies some finalizers automatically, but you can also specify your own.How finalizers workWhen you create a resource using a manifest file, you can specify finalizers in the metadata.finalizers field. When you attempt to delete the resource, the API server handling the delete request notices the values in the finalizers field and does the following:Modifies the object to add a metadata.deletionTimestamp field with the time you started the deletion.Prevents the object from being removed until all items are removed from its metadata.finalizers fieldReturns a 202 status code (HTTP "Accepted")The controller managing that finalizer notices the update to the object setting the metadata.deletionTimestamp, indicating deletion of the object has been requested. The controller then attempts to satisfy the requirements of the finalizers specified for that resource. Each time a finalizer condition is satisfied, the controller removes that key from the resource's finalizers field. When the finalizers field is emptied, an object with a deletionTimestamp field set is automatically deleted. You can also use finalizers to prevent deletion of unmanaged resources.A common example of a finalizer is kubernetes.io/pv-protection, which prevents accidental deletion of PersistentVolume objects. When a PersistentVolume object is in use by a Pod, Kubernetes adds the pv-protection finalizer. If you try to delete the PersistentVolume, it enters a Terminating status, but the controller can't delete it because the finalizer exists. When the Pod stops using the PersistentVolume, Kubernetes clears the pv-protection finalizer, and the controller deletes the volume.Note:When you DELETE an object, Kubernetes adds the deletion timestamp for that object and then immediately starts to restrict changes to the .metadata.finalizers field for the object that is now pending deletion. You can remove existing finalizers (deleting an entry from the finalizers list) but you cannot add a new finalizer. You also cannot modify the deletionTimestamp for an object once it is set.After the deletion is requested, you can not resurrect this object. The only way is to delete it and make a new similar object.Note:Custom finalizer names must be publicly qualified finalizer names, such as example.com/finalizer-name. Kubernetes enforces this format; the API server rejects writes to objects where the change does not use qualified finalizer names for any custom finalizer.Owner references, labels, and finalizersLike labels, owner references describe the relationships between objects in Kubernetes, but are used for a different purpose. When a controller manages objects like Pods, it uses labels to track changes to groups of related objects. For example, when a Job creates one or more Pods, the Job controller applies labels to those pods and tracks changes to any Pods in the cluster with the same label.The Job controller also adds owner references to those Pods, pointing at the Job that created the Pods. If you delete the Job while these Pods are running, Kubernetes uses the owner references (not labels) to determine which Pods in the cluster need cleanup.Kubernetes also processes finalizers when it identifies owner references on a resource targeted for deletion.In some situations, finalizers can block the deletion of dependent objects, which can cause the targeted owner object to remain for longer than expected without being fully deleted. In these situations, you should check finalizers and owner references on the target owner and dependent objects to troubleshoot the cause.Note:In cases where objects are stuck in a deleting state, avoid manually removing finalizers to allow deletion to continue. Finalizers are usually added to resources for a reason, so forcefully removing them can lead to issues in your cluster. This should only be done when the purpose of the finalizer is understood and is accomplished in another way (for example, manually cleaning up some dependent object).What's nextRead Using Finalizers to Control Deletion on the Kubernetes blog.

Finalizers


ResourceClaimResourceClaim describes a request for access to resources in the cluster, for use by workloads.apiVersion: resource.k8s.io/v1import "k8s.io/api/resource/v1"ResourceClaimResourceClaim describes a request for access to resources in the cluster, for use by workloads. For example, if a workload needs an accelerator device with specific properties, this is how that request is expressed. The status stanza tracks whether this claim has been satisfied and what specific resources have been allocated.This is an alpha type and requires enabling the DynamicResourceAllocation feature gate.apiVersion: resource.k8s.io/v1kind: ResourceClaimmetadata (ObjectMeta)Standard object metadataspec (ResourceClaimSpec), requiredSpec describes what is being requested and how to configure it. The spec is immutable.status (ResourceClaimStatus)Status describes whether the claim is ready to use and what has been allocated.ResourceClaimSpecResourceClaimSpec defines what is being requested in a ResourceClaim and how to configure it.devices (DeviceClaim)Devices defines how to request devices.DeviceClaim defines how to request devices with a ResourceClaim.devices.config ([]DeviceClaimConfiguration)Atomic: will be replaced during a mergeThis field holds configuration for multiple potential drivers which could satisfy requests in this claim. It is ignored while allocating the claim.DeviceClaimConfiguration is used for configuration parameters in DeviceClaim.devices.config.opaque (OpaqueDeviceConfiguration)Opaque provides driver-specific configuration parameters.OpaqueDeviceConfiguration contains configuration parameters for a driver in a format defined by the driver vendor.devices.config.opaque.driver (string), requiredDriver is used to determine which kubelet plugin needs to be passed these configuration parameters.An admission policy provided by the driver developer could use this to decide whether it needs to validate them.Must be a DNS subdomain and should end with a DNS domain owned by the vendor of the driver.devices.config.opaque.parameters (RawExtension), requiredParameters can contain arbitrary data. It is the responsibility of the driver developer to handle validation and versioning. Typically this includes self-identification and a version ("kind" + "apiVersion" for Kubernetes types), with conversion between different versions.The length of the raw data must be smaller or equal to 10 Ki.*RawExtension is used to hold extensions in external versions.To use this, make a field which has RawExtension as its type in your external, versioned struct, and Object in your internal struct. You also need to register your various plugin types.// Internal package:type MyAPIObject struct { runtime.TypeMeta `json:",inline"` MyPlugin runtime.Object `json:"myPlugin"` } type PluginA struct { AOption string `json:"aOption"` } // External package:type MyAPIObject struct { runtime.TypeMeta `json:",inline"` MyPlugin runtime.RawExtension `json:"myPlugin"` } type PluginA struct { AOption string `json:"aOption"` } // On the wire, the JSON will look something like this:{ "kind":"MyAPIObject", "apiVersion":"v1", "myPlugin": { "kind":"PluginA", "aOption":"foo", }, } So what happens? Decode first uses json or yaml to unmarshal the serialized data into your external MyAPIObject. That causes the raw JSON to be stored, but not unpacked. The next step is to copy (using pkg/conversion) into the internal struct. The runtime package's DefaultScheme has conversion functions installed which will unpack the JSON stored in RawExtension, turning it into the correct object type, and storing it in the Object. (TODO: In the case where the object is of an unknown type, a runtime.Unknown object will be created and stored.)*devices.config.requests ([]string)Atomic: will be replaced during a mergeRequests lists the names of requests where the configuration applies. If empty, it applies to all requests.References to subrequests must include the name of the main request and may include the subrequest using the format <main request>[/<subrequest>]. If just the main request is given, the configuration applies to all subrequests.devices.constraints ([]DeviceConstraint)Atomic: will be replaced during a mergeThese constraints must be satisfied by the set of devices that get allocated for the claim.DeviceConstraint must have exactly one field set besides Requests.devices.constraints.distinctAttribute (string)DistinctAttribute requires that all devices in question have this attribute and that its type and value are unique across those devices.This acts as the inverse of MatchAttribute.This constraint is used to avoid allocating multiple requests to the same device by ensuring attribute-level differentiation.This is useful for scenarios where resource requests must be fulfilled by separate physical devices. For example, a container requests two network interfaces that must be allocated from two different physical NICs.devices.constraints.matchAttribute (string)MatchAttribute requires that all devices in question have this attribute and that its type and value are the same across those devices.For example, if you specified "dra.example.com/numa" (a hypothetical example!), then only devices in the same NUMA node will be chosen. A device which does not have that attribute will not be chosen. All devices should use a value of the same type for this attribute because that is part of its specification, but if one device doesn't, then it also will not be chosen.Must include the domain qualifier.devices.constraints.requests ([]string)Atomic: will be replaced during a mergeRequests is a list of the one or more requests in this claim which must co-satisfy this constraint. If a request is fulfilled by multiple devices, then all of the devices must satisfy the constraint. If this is not specified, this constraint applies to all requests in this claim.References to subrequests must include the name of the main request and may include the subrequest using the format <main request>[/<subrequest>]. If just the main request is given, the constraint applies to all subrequests.devices.requests ([]DeviceRequest)Atomic: will be replaced during a mergeRequests represent individual requests for distinct devices which must all be satisfied. If empty, nothing needs to be allocated.DeviceRequest is a request for devices required for a claim. This is typically a request for a single resource like a device, but can also ask for several identical devices. With FirstAvailable it is also possible to provide a prioritized list of requests.devices.requests.name (string), requiredName can be used to reference this request in a pod.spec.containers[].resources.claims entry and in a constraint of the claim.References using the name in the DeviceRequest will uniquely identify a request when the Exactly field is set. When the FirstAvailable field is set, a reference to the name of the DeviceRequest will match whatever subrequest is chosen by the scheduler.Must be a DNS label.devices.requests.exactly (ExactDeviceRequest)Exactly specifies the details for a single request that must be met exactly for the request to be satisfied.One of Exactly or FirstAvailable must be set.ExactDeviceRequest is a request for one or more identical devices.devices.requests.exactly.deviceClassName (string), requiredDeviceClassName references a specific DeviceClass, which can define additional configuration and selectors to be inherited by this request.A DeviceClassName is required.Administrators may use this to restrict which devices may get requested by only installing classes with selectors for permitted devices. If users are free to request anything without restrictions, then administrators can create an empty DeviceClass for users to reference.devices.requests.exactly.adminAccess (boolean)AdminAccess indicates that this is a claim for administrative access to the device(s). Claims with AdminAccess are expected to be used for monitoring or other management services for a device. They ignore all ordinary claims to the device with respect to access modes and any resource allocations.This is an alpha field and requires enabling the DRAAdminAccess feature gate. Admin access is disabled if this field is unset or set to false, otherwise it is enabled.devices.requests.exactly.allocationMode (string)AllocationMode and its related fields define how devices are allocated to satisfy this request. Supported values are:ExactCount: This request is for a specific number of devices. This is the default. The exact number is provided in the count field.All: This request is for all of the matching devices in a pool. At least one device must exist on the node for the allocation to succeed. Allocation will fail if some devices are already allocated, unless adminAccess is requested.If AllocationMode is not specified, the default mode is ExactCount. If the mode is ExactCount and count is not specified, the default count is one. Any other requests must specify this field.More modes may get added in the future. Clients must refuse to handle requests with unknown modes.devices.requests.exactly.capacity (CapacityRequirements)Capacity define resource requirements against each capacity.If this field is unset and the device supports multiple allocations, the default value will be applied to each capacity according to requestPolicy. For the capacity that has no requestPolicy, default is the full capacity value.Applies to each device allocation. If Count > 1, the request fails if there aren't enough devices that meet the requirements. If AllocationMode is set to All, the request fails if there are devices that otherwise match the request, and have this capacity, with a value >= the requested amount, but which cannot be allocated to this request.CapacityRequirements defines the capacity requirements for a specific device request.devices.requests.exactly.capacity.requests (map[string]Quantity)Requests represent individual device resource requests for distinct resources, all of which must be provided by the device.This value is used as an additional filtering condition against the available capacity on the device. This is semantically equivalent to a CEL selector with device.capacity[\<domain>].\<name>.compareTo(quantity(\<request quantity>)) >= 0. For example, device.capacity['test-driver.cdi.k8s.io'].counters.compareTo(quantity('2')) >= 0.When a requestPolicy is defined, the requested amount is adjusted upward to the nearest valid value based on the policy. If the requested amount cannot be adjusted to a valid value—because it exceeds what the requestPolicy allows— the device is considered ineligible for allocation.For any capacity that is not explicitly requested: - If no requestPolicy is set, the default consumed capacity is equal to the full device capacity (i.e., the whole device is claimed).If a requestPolicy is set, the default consumed capacity is determined according to that policy.If the device allows multiple allocation, the aggregated amount across all requests must not exceed the capacity value. The consumed capacity, which may be adjusted based on the requestPolicy if defined, is recorded in the resource claim’s status.devices[*].consumedCapacity field.devices.requests.exactly.count (int64)Count is used only when the count mode is "ExactCount". Must be greater than zero. If AllocationMode is ExactCount and this field is not specified, the default is one.devices.requests.exactly.selectors ([]DeviceSelector)Atomic: will be replaced during a mergeSelectors define criteria which must be satisfied by a specific device in order for that device to be considered for this request. All selectors must be satisfied for a device to be considered.DeviceSelector must have exactly one field set.devices.requests.exactly.selectors.cel (CELDeviceSelector)CEL contains a CEL expression for selecting a device.CELDeviceSelector contains a CEL expression for selecting a device.devices.requests.exactly.selectors.cel.expression (string), requiredExpression is a CEL expression which evaluates a single device. It must evaluate to true when the device under consideration satisfies the desired criteria, and false when it does not. Any other result is an error and causes allocation of devices to abort.The expression's input is an object named "device", which carries the following properties:driver (string): the name of the driver which defines this device.attributes (map[string]object): the device's attributes, grouped by prefix (e.g. device.attributes["dra.example.com"] evaluates to an object with all of the attributes which were prefixed by "dra.example.com".capacity (map[string]object): the device's capacities, grouped by prefix.allowMultipleAllocations (bool): the allowMultipleAllocations property of the device (v1.34+ with the DRAConsumableCapacity feature enabled).Example: Consider a device with driver="dra.example.com", which exposes two attributes named "model" and "ext.example.com/family" and which exposes one capacity named "modules". This input to this expression would have the following fields:device.driver device.attributes["dra.example.com"].model device.attributes["ext.example.com"].family device.capacity["dra.example.com"].modules The device.driver field can be used to check for a specific driver, either as a high-level precondition (i.e. you only want to consider devices from this driver) or as part of a multi-clause expression that is meant to consider devices from different drivers.The value type of each attribute is defined by the device definition, and users who write these expressions must consult the documentation for their specific drivers. The value type of each capacity is Quantity.If an unknown prefix is used as a lookup in either device.attributes or device.capacity, an empty map will be returned. Any reference to an unknown field will cause an evaluation error and allocation to abort.A robust expression should check for the existence of attributes before referencing them.For ease of use, the cel.bind() function is enabled, and can be used to simplify expressions that access multiple attributes with the same domain. For example:cel.bind(dra, device.attributes["dra.example.com"], dra.someBool && dra.anotherBool) The length of the expression must be smaller or equal to 10 Ki. The cost of evaluating it is also limited based on the estimated number of logical steps.devices.requests.exactly.tolerations ([]DeviceToleration)Atomic: will be replaced during a mergeIf specified, the request's tolerations.Tolerations for NoSchedule are required to allocate a device which has a taint with that effect. The same applies to NoExecute.In addition, should any of the allocated devices get tainted with NoExecute after allocation and that effect is not tolerated, then all pods consuming the ResourceClaim get deleted to evict them. The scheduler will not let new pods reserve the claim while it has these tainted devices. Once all pods are evicted, the claim will get deallocated.The maximum number of tolerations is 16.This is an alpha field and requires enabling the DRADeviceTaints feature gate.The ResourceClaim this DeviceToleration is attached to tolerates any taint that matches the triple <key,value,effect> using the matching operator .devices.requests.exactly.tolerations.effect (string)Effect indicates the taint effect to match. Empty means match all taint effects. When specified, allowed values are NoSchedule and NoExecute.devices.requests.exactly.tolerations.key (string)Key is the taint key that the toleration applies to. Empty means match all taint keys. If the key is empty, operator must be Exists; this combination means to match all values and all keys. Must be a label name.devices.requests.exactly.tolerations.operator (string)Operator represents a key's relationship to the value. Valid operators are Exists and Equal. Defaults to Equal. Exists is equivalent to wildcard for value, so that a ResourceClaim can tolerate all taints of a particular category.devices.requests.exactly.tolerations.tolerationSeconds (int64)TolerationSeconds represents the period of time the toleration (which must be of effect NoExecute, otherwise this field is ignored) tolerates the taint. By default, it is not set, which means tolerate the taint forever (do not evict). Zero and negative values will be treated as 0 (evict immediately) by the system. If larger than zero, the time when the pod needs to be evicted is calculated as <time when taint was added> + <toleration seconds>.devices.requests.exactly.tolerations.value (string)Value is the taint value the toleration matches to. If the operator is Exists, the value must be empty, otherwise just a regular string. Must be a label value.devices.requests.firstAvailable ([]DeviceSubRequest)Atomic: will be replaced during a mergeFirstAvailable contains subrequests, of which exactly one will be selected by the scheduler. It tries to satisfy them in the order in which they are listed here. So if there are two entries in the list, the scheduler will only check the second one if it determines that the first one can not be used.DRA does not yet implement scoring, so the scheduler will select the first set of devices that satisfies all the requests in the claim. And if the requirements can be satisfied on more than one node, other scheduling features will determine which node is chosen. This means that the set of devices allocated to a claim might not be the optimal set available to the cluster. Scoring will be implemented later.*DeviceSubRequest describes a request for device provided in the claim.spec.devices.requests[].firstAvailable array. Each is typically a request for a single resource like a device, but can also ask for several identical devices.DeviceSubRequest is similar to ExactDeviceRequest, but doesn't expose the AdminAccess field as that one is only supported when requesting a specific device.*devices.requests.firstAvailable.deviceClassName (string), requiredDeviceClassName references a specific DeviceClass, which can define additional configuration and selectors to be inherited by this subrequest.A class is required. Which classes are available depends on the cluster.Administrators may use this to restrict which devices may get requested by only installing classes with selectors for permitted devices. If users are free to request anything without restrictions, then administrators can create an empty DeviceClass for users to reference.devices.requests.firstAvailable.name (string), requiredName can be used to reference this subrequest in the list of constraints or the list of configurations for the claim. References must use the format <main request>/<subrequest>.Must be a DNS label.devices.requests.firstAvailable.allocationMode (string)AllocationMode and its related fields define how devices are allocated to satisfy this subrequest. Supported values are:ExactCount: This request is for a specific number of devices. This is the default. The exact number is provided in the count field.All: This subrequest is for all of the matching devices in a pool. Allocation will fail if some devices are already allocated, unless adminAccess is requested.If AllocationMode is not specified, the default mode is ExactCount. If the mode is ExactCount and count is not specified, the default count is one. Any other subrequests must specify this field.More modes may get added in the future. Clients must refuse to handle requests with unknown modes.devices.requests.firstAvailable.capacity (CapacityRequirements)Capacity define resource requirements against each capacity.If this field is unset and the device supports multiple allocations, the default value will be applied to each capacity according to requestPolicy. For the capacity that has no requestPolicy, default is the full capacity value.Applies to each device allocation. If Count > 1, the request fails if there aren't enough devices that meet the requirements. If AllocationMode is set to All, the request fails if there are devices that otherwise match the request, and have this capacity, with a value >= the requested amount, but which cannot be allocated to this request.CapacityRequirements defines the capacity requirements for a specific device request.devices.requests.firstAvailable.capacity.requests (map[string]Quantity)Requests represent individual device resource requests for distinct resources, all of which must be provided by the device.This value is used as an additional filtering condition against the available capacity on the device. This is semantically equivalent to a CEL selector with device.capacity[\<domain>].\<name>.compareTo(quantity(\<request quantity>)) >= 0. For example, device.capacity['test-driver.cdi.k8s.io'].counters.compareTo(quantity('2')) >= 0.When a requestPolicy is defined, the requested amount is adjusted upward to the nearest valid value based on the policy. If the requested amount cannot be adjusted to a valid value—because it exceeds what the requestPolicy allows— the device is considered ineligible for allocation.For any capacity that is not explicitly requested: - If no requestPolicy is set, the default consumed capacity is equal to the full device capacity (i.e., the whole device is claimed).If a requestPolicy is set, the default consumed capacity is determined according to that policy.If the device allows multiple allocation, the aggregated amount across all requests must not exceed the capacity value. The consumed capacity, which may be adjusted based on the requestPolicy if defined, is recorded in the resource claim’s status.devices[*].consumedCapacity field.devices.requests.firstAvailable.count (int64)Count is used only when the count mode is "ExactCount". Must be greater than zero. If AllocationMode is ExactCount and this field is not specified, the default is one.devices.requests.firstAvailable.selectors ([]DeviceSelector)Atomic: will be replaced during a mergeSelectors define criteria which must be satisfied by a specific device in order for that device to be considered for this subrequest. All selectors must be satisfied for a device to be considered.DeviceSelector must have exactly one field set.devices.requests.firstAvailable.selectors.cel (CELDeviceSelector)CEL contains a CEL expression for selecting a device.CELDeviceSelector contains a CEL expression for selecting a device.devices.requests.firstAvailable.selectors.cel.expression (string), requiredExpression is a CEL expression which evaluates a single device. It must evaluate to true when the device under consideration satisfies the desired criteria, and false when it does not. Any other result is an error and causes allocation of devices to abort.The expression's input is an object named "device", which carries the following properties:driver (string): the name of the driver which defines this device.attributes (map[string]object): the device's attributes, grouped by prefix (e.g. device.attributes["dra.example.com"] evaluates to an object with all of the attributes which were prefixed by "dra.example.com".capacity (map[string]object): the device's capacities, grouped by prefix.allowMultipleAllocations (bool): the allowMultipleAllocations property of the device (v1.34+ with the DRAConsumableCapacity feature enabled).Example: Consider a device with driver="dra.example.com", which exposes two attributes named "model" and "ext.example.com/family" and which exposes one capacity named "modules". This input to this expression would have the following fields:device.driver device.attributes["dra.example.com"].model device.attributes["ext.example.com"].family device.capacity["dra.example.com"].modules The device.driver field can be used to check for a specific driver, either as a high-level precondition (i.e. you only want to consider devices from this driver) or as part of a multi-clause expression that is meant to consider devices from different drivers.The value type of each attribute is defined by the device definition, and users who write these expressions must consult the documentation for their specific drivers. The value type of each capacity is Quantity.If an unknown prefix is used as a lookup in either device.attributes or device.capacity, an empty map will be returned. Any reference to an unknown field will cause an evaluation error and allocation to abort.A robust expression should check for the existence of attributes before referencing them.For ease of use, the cel.bind() function is enabled, and can be used to simplify expressions that access multiple attributes with the same domain. For example:cel.bind(dra, device.attributes["dra.example.com"], dra.someBool && dra.anotherBool) The length of the expression must be smaller or equal to 10 Ki. The cost of evaluating it is also limited based on the estimated number of logical steps.devices.requests.firstAvailable.tolerations ([]DeviceToleration)Atomic: will be replaced during a mergeIf specified, the request's tolerations.Tolerations for NoSchedule are required to allocate a device which has a taint with that effect. The same applies to NoExecute.In addition, should any of the allocated devices get tainted with NoExecute after allocation and that effect is not tolerated, then all pods consuming the ResourceClaim get deleted to evict them. The scheduler will not let new pods reserve the claim while it has these tainted devices. Once all pods are evicted, the claim will get deallocated.The maximum number of tolerations is 16.This is an alpha field and requires enabling the DRADeviceTaints feature gate.The ResourceClaim this DeviceToleration is attached to tolerates any taint that matches the triple <key,value,effect> using the matching operator .devices.requests.firstAvailable.tolerations.effect (string)Effect indicates the taint effect to match. Empty means match all taint effects. When specified, allowed values are NoSchedule and NoExecute.devices.requests.firstAvailable.tolerations.key (string)Key is the taint key that the toleration applies to. Empty means match all taint keys. If the key is empty, operator must be Exists; this combination means to match all values and all keys. Must be a label name.devices.requests.firstAvailable.tolerations.operator (string)Operator represents a key's relationship to the value. Valid operators are Exists and Equal. Defaults to Equal. Exists is equivalent to wildcard for value, so that a ResourceClaim can tolerate all taints of a particular category.devices.requests.firstAvailable.tolerations.tolerationSeconds (int64)TolerationSeconds represents the period of time the toleration (which must be of effect NoExecute, otherwise this field is ignored) tolerates the taint. By default, it is not set, which means tolerate the taint forever (do not evict). Zero and negative values will be treated as 0 (evict immediately) by the system. If larger than zero, the time when the pod needs to be evicted is calculated as <time when taint was added> + <toleration seconds>.devices.requests.firstAvailable.tolerations.value (string)Value is the taint value the toleration matches to. If the operator is Exists, the value must be empty, otherwise just a regular string. Must be a label value.ResourceClaimStatusResourceClaimStatus tracks whether the resource has been allocated and what the result of that was.allocation (AllocationResult)Allocation is set once the claim has been allocated successfully.AllocationResult contains attributes of an allocated resource.allocation.allocationTimestamp (Time)AllocationTimestamp stores the time when the resources were allocated. This field is not guaranteed to be set, in which case that time is unknown.This is an alpha field and requires enabling the DRADeviceBindingConditions and DRAResourceClaimDeviceStatus feature gate.Time is a wrapper around time.Time which supports correct marshaling to YAML and JSON. Wrappers are provided for many of the factory methods that the time package offers.allocation.devices (DeviceAllocationResult)Devices is the result of allocating devices.DeviceAllocationResult is the result of allocating devices.allocation.devices.config ([]DeviceAllocationConfiguration)Atomic: will be replaced during a mergeThis field is a combination of all the claim and class configuration parameters. Drivers can distinguish between those based on a flag.This includes configuration parameters for drivers which have no allocated devices in the result because it is up to the drivers which configuration parameters they support. They can silently ignore unknown configuration parameters.DeviceAllocationConfiguration gets embedded in an AllocationResult.allocation.devices.config.source (string), requiredSource records whether the configuration comes from a class and thus is not something that a normal user would have been able to set or from a claim.allocation.devices.config.opaque (OpaqueDeviceConfiguration)Opaque provides driver-specific configuration parameters.OpaqueDeviceConfiguration contains configuration parameters for a driver in a format defined by the driver vendor.allocation.devices.config.opaque.driver (string), requiredDriver is used to determine which kubelet plugin needs to be passed these configuration parameters.An admission policy provided by the driver developer could use this to decide whether it needs to validate them.Must be a DNS subdomain and should end with a DNS domain owned by the vendor of the driver.allocation.devices.config.opaque.parameters (RawExtension), requiredParameters can contain arbitrary data. It is the responsibility of the driver developer to handle validation and versioning. Typically this includes self-identification and a version ("kind" + "apiVersion" for Kubernetes types), with conversion between different versions.The length of the raw data must be smaller or equal to 10 Ki.*RawExtension is used to hold extensions in external versions.To use this, make a field which has RawExtension as its type in your external, versioned struct, and Object in your internal struct. You also need to register your various plugin types.// Internal package:type MyAPIObject struct { runtime.TypeMeta json:",inline" MyPlugin runtime.Object json:"myPlugin" }type PluginA struct { AOption string json:"aOption" }// External package:type MyAPIObject struct { runtime.TypeMeta json:",inline" MyPlugin runtime.RawExtension json:"myPlugin" }type PluginA struct { AOption string json:"aOption" }// On the wire, the JSON will look something like this:{ "kind":"MyAPIObject", "apiVersion":"v1", "myPlugin": { "kind":"PluginA", "aOption":"foo", }, }So what happens? Decode first uses json or yaml to unmarshal the serialized data into your external MyAPIObject. That causes the raw JSON to be stored, but not unpacked. The next step is to copy (using pkg/conversion) into the internal struct. The runtime package's DefaultScheme has conversion functions installed which will unpack the JSON stored in RawExtension, turning it into the correct object type, and storing it in the Object. (TODO: In the case where the object is of an unknown type, a runtime.Unknown object will be created and stored.)*allocation.devices.config.requests ([]string)Atomic: will be replaced during a mergeRequests lists the names of requests where the configuration applies. If empty, its applies to all requests.References to subrequests must include the name of the main request and may include the subrequest using the format <main request>[/<subrequest>]. If just the main request is given, the configuration applies to all subrequests.allocation.devices.results ([]DeviceRequestAllocationResult)Atomic: will be replaced during a mergeResults lists all allocated devices.DeviceRequestAllocationResult contains the allocation result for one request.allocation.devices.results.device (string), requiredDevice references one device instance via its name in the driver's resource pool. It must be a DNS label.allocation.devices.results.driver (string), requiredDriver specifies the name of the DRA driver whose kubelet plugin should be invoked to process the allocation once the claim is needed on a node.Must be a DNS subdomain and should end with a DNS domain owned by the vendor of the driver.allocation.devices.results.pool (string), requiredThis name together with the driver name and the device name field identify which device was allocated (\<driver name>/\<pool name>/\<device name>).Must not be longer than 253 characters and may contain one or more DNS sub-domains separated by slashes.allocation.devices.results.request (string), requiredRequest is the name of the request in the claim which caused this device to be allocated. If it references a subrequest in the firstAvailable list on a DeviceRequest, this field must include both the name of the main request and the subrequest using the format <main request>/<subrequest>.Multiple devices may have been allocated per request.allocation.devices.results.adminAccess (boolean)AdminAccess indicates that this device was allocated for administrative access. See the corresponding request field for a definition of mode.This is an alpha field and requires enabling the DRAAdminAccess feature gate. Admin access is disabled if this field is unset or set to false, otherwise it is enabled.allocation.devices.results.bindingConditions ([]string)Atomic: will be replaced during a mergeBindingConditions contains a copy of the BindingConditions from the corresponding ResourceSlice at the time of allocation.This is an alpha field and requires enabling the DRADeviceBindingConditions and DRAResourceClaimDeviceStatus feature gates.allocation.devices.results.bindingFailureConditions ([]string)Atomic: will be replaced during a mergeBindingFailureConditions contains a copy of the BindingFailureConditions from the corresponding ResourceSlice at the time of allocation.This is an alpha field and requires enabling the DRADeviceBindingConditions and DRAResourceClaimDeviceStatus feature gates.allocation.devices.results.consumedCapacity (map[string]Quantity)ConsumedCapacity tracks the amount of capacity consumed per device as part of the claim request. The consumed amount may differ from the requested amount: it is rounded up to the nearest valid value based on the device’s requestPolicy if applicable (i.e., may not be less than the requested amount).The total consumed capacity for each device must not exceed the DeviceCapacity's Value.This field is populated only for devices that allow multiple allocations. All capacity entries are included, even if the consumed amount is zero.allocation.devices.results.shareID (string)ShareID uniquely identifies an individual allocation share of the device, used when the device supports multiple simultaneous allocations. It serves as an additional map key to differentiate concurrent shares of the same device.allocation.devices.results.tolerations ([]DeviceToleration)Atomic: will be replaced during a mergeA copy of all tolerations specified in the request at the time when the device got allocated.The maximum number of tolerations is 16.This is an alpha field and requires enabling the DRADeviceTaints feature gate.The ResourceClaim this DeviceToleration is attached to tolerates any taint that matches the triple <key,value,effect> using the matching operator .allocation.devices.results.tolerations.effect (string)Effect indicates the taint effect to match. Empty means match all taint effects. When specified, allowed values are NoSchedule and NoExecute.allocation.devices.results.tolerations.key (string)Key is the taint key that the toleration applies to. Empty means match all taint keys. If the key is empty, operator must be Exists; this combination means to match all values and all keys. Must be a label name.allocation.devices.results.tolerations.operator (string)Operator represents a key's relationship to the value. Valid operators are Exists and Equal. Defaults to Equal. Exists is equivalent to wildcard for value, so that a ResourceClaim can tolerate all taints of a particular category.allocation.devices.results.tolerations.tolerationSeconds (int64)TolerationSeconds represents the period of time the toleration (which must be of effect NoExecute, otherwise this field is ignored) tolerates the taint. By default, it is not set, which means tolerate the taint forever (do not evict). Zero and negative values will be treated as 0 (evict immediately) by the system. If larger than zero, the time when the pod needs to be evicted is calculated as <time when taint was added> + <toleration seconds>.allocation.devices.results.tolerations.value (string)Value is the taint value the toleration matches to. If the operator is Exists, the value must be empty, otherwise just a regular string. Must be a label value.allocation.nodeSelector (NodeSelector)NodeSelector defines where the allocated resources are available. If unset, they are available everywhere.A node selector represents the union of the results of one or more label queries over a set of nodes; that is, it represents the OR of the selectors represented by the node selector terms.allocation.nodeSelector.nodeSelectorTerms ([]NodeSelectorTerm), requiredAtomic: will be replaced during a mergeRequired. A list of node selector terms. The terms are ORed.A null or empty node selector term matches no objects. The requirements of them are ANDed. The TopologySelectorTerm type implements a subset of the NodeSelectorTerm.allocation.nodeSelector.nodeSelectorTerms.matchExpressions ([]NodeSelectorRequirement)Atomic: will be replaced during a mergeA list of node selector requirements by node's labels.allocation.nodeSelector.nodeSelectorTerms.matchFields ([]NodeSelectorRequirement)Atomic: will be replaced during a mergeA list of node selector requirements by node's fields.devices ([]AllocatedDeviceStatus)Map: unique values on keys driver, device, pool, shareID will be kept during a mergeDevices contains the status of each device allocated for this claim, as reported by the driver. This can include driver-specific information. Entries are owned by their respective drivers.*AllocatedDeviceStatus contains the status of an allocated device, if the driver chooses to report it. This may include driver-specific information.The combination of Driver, Pool, Device, and ShareID must match the corresponding key in Status.Allocation.Devices.*devices.device (string), requiredDevice references one device instance via its name in the driver's resource pool. It must be a DNS label.devices.driver (string), requiredDriver specifies the name of the DRA driver whose kubelet plugin should be invoked to process the allocation once the claim is needed on a node.Must be a DNS subdomain and should end with a DNS domain owned by the vendor of the driver.devices.pool (string), requiredThis name together with the driver name and the device name field identify which device was allocated (\<driver name>/\<pool name>/\<device name>).Must not be longer than 253 characters and may contain one or more DNS sub-domains separated by slashes.devices.conditions ([]Condition)Map: unique values on key type will be kept during a mergeConditions contains the latest observation of the device's state. If the device has been configured according to the class and claim config references, the Ready condition should be True.Must not contain more than 8 entries.Condition contains details for one aspect of the current state of this API Resource.devices.conditions.lastTransitionTime (Time), requiredlastTransitionTime is the last time the condition transitioned from one status to another. This should be when the underlying condition changed. If that is not known, then using the time when the API field changed is acceptable.Time is a wrapper around time.Time which supports correct marshaling to YAML and JSON. Wrappers are provided for many of the factory methods that the time package offers.devices.conditions.message (string), requiredmessage is a human readable message indicating details about the transition. This may be an empty string.devices.conditions.reason (string), requiredreason contains a programmatic identifier indicating the reason for the condition's last transition. Producers of specific condition types may define expected values and meanings for this field, and whether the values are considered a guaranteed API. The value should be a CamelCase string. This field may not be empty.devices.conditions.status (string), requiredstatus of the condition, one of True, False, Unknown.devices.conditions.type (string), requiredtype of condition in CamelCase or in foo.example.com/CamelCase.devices.conditions.observedGeneration (int64)observedGeneration represents the .metadata.generation that the condition was set based upon. For instance, if .metadata.generation is currently 12, but the .status.conditions[x].observedGeneration is 9, the condition is out of date with respect to the current state of the instance.devices.data (RawExtension)Data contains arbitrary driver-specific data.The length of the raw data must be smaller or equal to 10 Ki.*RawExtension is used to hold extensions in external versions.To use this, make a field which has RawExtension as its type in your external, versioned struct, and Object in your internal struct. You also need to register your various plugin types.// Internal package:type MyAPIObject struct { runtime.TypeMeta `json:",inline"` MyPlugin runtime.Object `json:"myPlugin"` } type PluginA struct { AOption string `json:"aOption"` } // External package:type MyAPIObject struct { runtime.TypeMeta `json:",inline"` MyPlugin runtime.RawExtension `json:"myPlugin"` } type PluginA struct { AOption string `json:"aOption"` } // On the wire, the JSON will look something like this:{ "kind":"MyAPIObject", "apiVersion":"v1", "myPlugin": { "kind":"PluginA", "aOption":"foo", }, } So what happens? Decode first uses json or yaml to unmarshal the serialized data into your external MyAPIObject. That causes the raw JSON to be stored, but not unpacked. The next step is to copy (using pkg/conversion) into the internal struct. The runtime package's DefaultScheme has conversion functions installed which will unpack the JSON stored in RawExtension, turning it into the correct object type, and storing it in the Object. (TODO: In the case where the object is of an unknown type, a runtime.Unknown object will be created and stored.)*devices.networkData (NetworkDeviceData)NetworkData contains network-related information specific to the device.NetworkDeviceData provides network-related details for the allocated device. This information may be filled by drivers or other components to configure or identify the device within a network context.devices.networkData.hardwareAddress (string)HardwareAddress represents the hardware address (e.g. MAC Address) of the device's network interface.Must not be longer than 128 characters.devices.networkData.interfaceName (string)InterfaceName specifies the name of the network interface associated with the allocated device. This might be the name of a physical or virtual network interface being configured in the pod.Must not be longer than 256 characters.devices.networkData.ips ([]string)Atomic: will be replaced during a mergeIPs lists the network addresses assigned to the device's network interface. This can include both IPv4 and IPv6 addresses. The IPs are in the CIDR notation, which includes both the address and the associated subnet mask. e.g.: "192.0.2.5/24" for IPv4 and "2001:db8::5/64" for IPv6.devices.shareID (string)ShareID uniquely identifies an individual allocation share of the device.reservedFor ([]ResourceClaimConsumerReference)Patch strategy: merge on key uidMap: unique values on key uid will be kept during a mergeReservedFor indicates which entities are currently allowed to use the claim. A Pod which references a ResourceClaim which is not reserved for that Pod will not be started. A claim that is in use or might be in use because it has been reserved must not get deallocated.In a cluster with multiple scheduler instances, two pods might get scheduled concurrently by different schedulers. When they reference the same ResourceClaim which already has reached its maximum number of consumers, only one pod can be scheduled.Both schedulers try to add their pod to the claim.status.reservedFor field, but only the update that reaches the API server first gets stored. The other one fails with an error and the scheduler which issued it knows that it must put the pod back into the queue, waiting for the ResourceClaim to become usable again.There can be at most 256 such reservations. This may get increased in the future, but not reduced.ResourceClaimConsumerReference contains enough information to let you locate the consumer of a ResourceClaim. The user must be a resource in the same namespace as the ResourceClaim.reservedFor.name (string), requiredName is the name of resource being referenced.reservedFor.resource (string), requiredResource is the type of resource being referenced, for example "pods".reservedFor.uid (string), requiredUID identifies exactly one incarnation of the resource.reservedFor.apiGroup (string)APIGroup is the group for the resource being referenced. It is empty for the core API. This matches the group in the APIVersion that is used when creating the resources.ResourceClaimListResourceClaimList is a collection of claims.apiVersion: resource.k8s.io/v1kind: ResourceClaimListmetadata (ListMeta)Standard list metadataitems ([]ResourceClaim), requiredItems is the list of resource claims.Operationsget read the specified ResourceClaimHTTP RequestGET /apis/resource.k8s.io/v1/namespaces/{namespace}/resourceclaims/{name}Parametersname (in path): string, requiredname of the ResourceClaimnamespace (in path): string, requirednamespacepretty (in query): stringprettyResponse200 (ResourceClaim): OK401: Unauthorizedget read status of the specified ResourceClaimHTTP RequestGET /apis/resource.k8s.io/v1/namespaces/{namespace}/resourceclaims/{name}/statusParametersname (in path): string, requiredname of the ResourceClaimnamespace (in path): string, requirednamespacepretty (in query): stringprettyResponse200 (ResourceClaim): OK401: Unauthorizedlist list or watch objects of kind ResourceClaimHTTP RequestGET /apis/resource.k8s.io/v1/namespaces/{namespace}/resourceclaimsParametersnamespace (in path): string, requirednamespaceallowWatchBookmarks (in query): booleanallowWatchBookmarkscontinue (in query): stringcontinuefieldSelector (in query): stringfieldSelectorlabelSelector (in query): stringlabelSelectorlimit (in query): integerlimitpretty (in query): stringprettyresourceVersion (in query): stringresourceVersionresourceVersionMatch (in query): stringresourceVersionMatchsendInitialEvents (in query): booleansendInitialEventstimeoutSeconds (in query): integertimeoutSecondswatch (in query): booleanwatchResponse200 (ResourceClaimList): OK401: Unauthorizedlist list or watch objects of kind ResourceClaimHTTP RequestGET /apis/resource.k8s.io/v1/resourceclaimsParametersallowWatchBookmarks (in query): booleanallowWatchBookmarkscontinue (in query): stringcontinuefieldSelector (in query): stringfieldSelectorlabelSelector (in query): stringlabelSelectorlimit (in query): integerlimitpretty (in query): stringprettyresourceVersion (in query): stringresourceVersionresourceVersionMatch (in query): stringresourceVersionMatchsendInitialEvents (in query): booleansendInitialEventstimeoutSeconds (in query): integertimeoutSecondswatch (in query): booleanwatchResponse200 (ResourceClaimList): OK401: Unauthorizedcreate create a ResourceClaimHTTP RequestPOST /apis/resource.k8s.io/v1/namespaces/{namespace}/resourceclaimsParametersnamespace (in path): string, requirednamespacebody: ResourceClaim, requireddryRun (in query): stringdryRunfieldManager (in query): stringfieldManagerfieldValidation (in query): stringfieldValidationpretty (in query): stringprettyResponse200 (ResourceClaim): OK201 (ResourceClaim): Created202 (ResourceClaim): Accepted401: Unauthorizedupdate replace the specified ResourceClaimHTTP RequestPUT /apis/resource.k8s.io/v1/namespaces/{namespace}/resourceclaims/{name}Parametersname (in path): string, requiredname of the ResourceClaimnamespace (in path): string, requirednamespacebody: ResourceClaim, requireddryRun (in query): stringdryRunfieldManager (in query): stringfieldManagerfieldValidation (in query): stringfieldValidationpretty (in query): stringprettyResponse200 (ResourceClaim): OK201 (ResourceClaim): Created401: Unauthorizedupdate replace status of the specified ResourceClaimHTTP RequestPUT /apis/resource.k8s.io/v1/namespaces/{namespace}/resourceclaims/{name}/statusParametersname (in path): string, requiredname of the ResourceClaimnamespace (in path): string, requirednamespacebody: ResourceClaim, requireddryRun (in query): stringdryRunfieldManager (in query): stringfieldManagerfieldValidation (in query): stringfieldValidationpretty (in query): stringprettyResponse200 (ResourceClaim): OK201 (ResourceClaim): Created401: Unauthorizedpatch partially update the specified ResourceClaimHTTP RequestPATCH /apis/resource.k8s.io/v1/namespaces/{namespace}/resourceclaims/{name}Parametersname (in path): string, requiredname of the ResourceClaimnamespace (in path): string, requirednamespacebody: Patch, requireddryRun (in query): stringdryRunfieldManager (in query): stringfieldManagerfieldValidation (in query): stringfieldValidationforce (in query): booleanforcepretty (in query): stringprettyResponse200 (ResourceClaim): OK201 (ResourceClaim): Created401: Unauthorizedpatch partially update status of the specified ResourceClaimHTTP RequestPATCH /apis/resource.k8s.io/v1/namespaces/{namespace}/resourceclaims/{name}/statusParametersname (in path): string, requiredname of the ResourceClaimnamespace (in path): string, requirednamespacebody: Patch, requireddryRun (in query): stringdryRunfieldManager (in query): stringfieldManagerfieldValidation (in query): stringfieldValidationforce (in query): booleanforcepretty (in query): stringprettyResponse200 (ResourceClaim): OK201 (ResourceClaim): Created401: Unauthorizeddelete delete a ResourceClaimHTTP RequestDELETE /apis/resource.k8s.io/v1/namespaces/{namespace}/resourceclaims/{name}Parametersname (in path): string, requiredname of the ResourceClaimnamespace (in path): string, requirednamespacebody: DeleteOptionsdryRun (in query): stringdryRungracePeriodSeconds (in query): integergracePeriodSecondsignoreStoreReadErrorWithClusterBreakingPotential (in query): booleanignoreStoreReadErrorWithClusterBreakingPotentialpretty (in query): stringprettypropagationPolicy (in query): stringpropagationPolicyResponse200 (ResourceClaim): OK202 (ResourceClaim): Accepted401: Unauthorizeddeletecollection delete collection of ResourceClaimHTTP RequestDELETE /apis/resource.k8s.io/v1/namespaces/{namespace}/resourceclaimsParametersnamespace (in path): string, requirednamespacebody: DeleteOptionscontinue (in query): stringcontinuedryRun (in query): stringdryRunfieldSelector (in query): stringfieldSelectorgracePeriodSeconds (in query): integergracePeriodSecondsignoreStoreReadErrorWithClusterBreakingPotential (in query): booleanignoreStoreReadErrorWithClusterBreakingPotentiallabelSelector (in query): stringlabelSelectorlimit (in query): integerlimitpretty (in query): stringprettypropagationPolicy (in query): stringpropagationPolicyresourceVersion (in query): stringresourceVersionresourceVersionMatch (in query): stringresourceVersionMatchsendInitialEvents (in query): booleansendInitialEventstimeoutSeconds (in query): integertimeoutSecondsResponse200 (Status): OK401: UnauthorizedThis page is automatically generated.If you plan to report an issue with this page, mention that the page is auto-generated in your issue description. The fix may need to happen elsewhere in the Kubernetes project.

ResourceClaim


Running Kubernetes Node Components as a Non-root UserFEATURE STATE: Kubernetes v1.22 [alpha]This document describes how to run Kubernetes Node components such as kubelet, CRI, OCI, and CNI without root privileges, by using a user namespace.This technique is also known as rootless mode.Note:This document describes how to run Kubernetes Node components (and hence pods) as a non-root user.If you are just looking for how to run a pod as a non-root user, see SecurityContext.Before you beginYour Kubernetes server must be at or later than version 1.22.To check the version, enter kubectl version.Enable Cgroup v2Enable systemd with user sessionConfigure several sysctl values, depending on host Linux distributionEnsure that your unprivileged user is listed in /etc/subuid and /etc/subgidEnable the KubeletInUserNamespace feature gateRunning Kubernetes inside Rootless Docker/Podmankindkind supports running Kubernetes inside Rootless Docker or Rootless Podman.See Running kind with Rootless Docker.minikubeminikube also supports running Kubernetes inside Rootless Docker or Rootless Podman.See the Minikube documentation:Rootless DockerRootless PodmanRunning Kubernetes inside Unprivileged ContainersNote: This section links to third party projects that provide functionality required by Kubernetes. The Kubernetes project authors aren't responsible for these projects, which are listed alphabetically. To add a project to this list, read the content guide before submitting a change. More information.sysboxSysbox is an open-source container runtime (similar to "runc") that supports running system-level workloads such as Docker and Kubernetes inside unprivileged containers isolated with the Linux user namespace.See Sysbox Quick Start Guide: Kubernetes-in-Docker for more info.Sysbox supports running Kubernetes inside unprivileged containers without requiring Cgroup v2 and without the KubeletInUserNamespace feature gate. It does this by exposing specially crafted /proc and /sys filesystems inside the container plus several other advanced OS virtualization techniques.Running Rootless Kubernetes directly on a hostNote: This section links to third party projects that provide functionality required by Kubernetes. The Kubernetes project authors aren't responsible for these projects, which are listed alphabetically. To add a project to this list, read the content guide before submitting a change. More information.K3sK3s experimentally supports rootless mode.See Running K3s with Rootless mode for the usage.UsernetesUsernetes is a reference distribution of Kubernetes that can be installed under $HOME directory without the root privilege.Usernetes supports both containerd and CRI-O as CRI runtimes. Usernetes supports multi-node clusters using Flannel (VXLAN).See the Usernetes repo for the usage.Manually deploy a node that runs the kubelet in a user namespaceThis section provides hints for running Kubernetes in a user namespace manually.Note:This section is intended to be read by developers of Kubernetes distributions, not by end users.Creating a user namespaceThe first step is to create a user namespace.If you are trying to run Kubernetes in a user-namespaced container such as Rootless Docker/Podman or LXC/LXD, you are all set, and you can go to the next subsection.Otherwise you have to create a user namespace by yourself, by calling unshare(2) with CLONE_NEWUSER.A user namespace can be also unshared by using command line tools such as:unshare(1)RootlessKitbecome-rootAfter unsharing the user namespace, you will also have to unshare other namespaces such as mount namespace.You do not need to call chroot() nor pivot_root() after unsharing the mount namespace, however, you have to mount writable filesystems on several directories in the namespace.At least, the following directories need to be writable in the namespace (not outside the namespace):/etc/run/var/logs/var/lib/kubelet/var/lib/cni/var/lib/containerd (for containerd)/var/lib/containers (for CRI-O)Creating a delegated cgroup treeIn addition to the user namespace, you also need to have a writable cgroup tree with cgroup v2.Note:Kubernetes support for running Node components in user namespaces requires cgroup v2. Cgroup v1 is not supported.If you are trying to run Kubernetes in Rootless Docker/Podman or LXC/LXD on a systemd-based host, you are all set.Otherwise you have to create a systemd unit with Delegate=yes property to delegate a cgroup tree with writable permission.On your node, systemd must already be configured to allow delegation; for more details, see cgroup v2 in the Rootless Containers documentation.Configuring networkNote: This section links to third party projects that provide functionality required by Kubernetes. The Kubernetes project authors aren't responsible for these projects, which are listed alphabetically. To add a project to this list, read the content guide before submitting a change. More information.The network namespace of the Node components has to have a non-loopback interface, which can be for example configured with slirp4netns, VPNKit, or lxc-user-nic(1).The network namespaces of the Pods can be configured with regular CNI plugins. For multi-node networking, Flannel (VXLAN, 8472/UDP) is known to work.Ports such as the kubelet port (10250/TCP) and NodePort service ports have to be exposed from the Node network namespace to the host with an external port forwarder, such as RootlessKit, slirp4netns, or socat(1).You can use the port forwarder from K3s. See Running K3s in Rootless Mode for more details. The implementation can be found in the pkg/rootlessports package of k3s.Configuring CRIThe kubelet relies on a container runtime. You should deploy a container runtime such as containerd or CRI-O and ensure that it is running within the user namespace before the kubelet starts.Running CRI plugin of containerd in a user namespace is supported since containerd 1.4.Running containerd within a user namespace requires the following configurations.version = 2 [plugins."io.containerd.grpc.v1.cri"] # Disable AppArmor disable_apparmor = true # Ignore an error during setting oom_score_adj restrict_oom_score_adj = true # Disable hugetlb cgroup v2 controller (because systemd does not support delegating hugetlb controller) disable_hugetlb_controller = true [plugins."io.containerd.grpc.v1.cri".containerd] # Using non-fuse overlayfs is also possible for kernel >= 5.11, but requires SELinux to be disabled snapshotter = "fuse-overlayfs" [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options] # We use cgroupfs that is delegated by systemd, so we do not use SystemdCgroup driver # (unless you run another systemd in the namespace) SystemdCgroup = false The default path of the configuration file is /etc/containerd/config.toml. The path can be specified with containerd -c /path/to/containerd/config.toml.Running CRI-O in a user namespace is supported since CRI-O 1.22.CRI-O requires an environment variable _CRIO_ROOTLESS=1 to be set.The following configurations are also recommended:[crio] storage_driver = "overlay" # Using non-fuse overlayfs is also possible for kernel >= 5.11, but requires SELinux to be disabled storage_option = ["overlay.mount_program=/usr/local/bin/fuse-overlayfs"] [crio.runtime] # We use cgroupfs that is delegated by systemd, so we do not use "systemd" driver # (unless you run another systemd in the namespace) cgroup_manager = "cgroupfs" The default path of the configuration file is /etc/crio/crio.conf. The path can be specified with crio --config /path/to/crio/crio.conf.Configuring kubeletRunning kubelet in a user namespace requires the following configuration:apiVersion: kubelet.config.k8s.io/v1beta1 kind: KubeletConfiguration featureGates: KubeletInUserNamespace: true # We use cgroupfs that is delegated by systemd, so we do not use "systemd" driver # (unless you run another systemd in the namespace) cgroupDriver: "cgroupfs" When the KubeletInUserNamespace feature gate is enabled, the kubelet ignores errors that may happen during setting the following sysctl values on the node.vm.overcommit_memoryvm.panic_on_oomkernel.panickernel.panic_on_oopskernel.keys.root_maxkeyskernel.keys.root_maxbytes.Within a user namespace, the kubelet also ignores any error raised from trying to open /dev/kmsg. This feature gate also allows kube-proxy to ignore an error during setting RLIMIT_NOFILE.The KubeletInUserNamespace feature gate was introduced in Kubernetes v1.22 with "alpha" status.Running kubelet in a user namespace without using this feature gate is also possible by mounting a specially crafted proc filesystem (as done by Sysbox), but not officially supported.Configuring kube-proxyRunning kube-proxy in a user namespace requires the following configuration:apiVersion: kubeproxy.config.k8s.io/v1alpha1 kind: KubeProxyConfiguration mode: "iptables" # or "userspace" conntrack: # Skip setting sysctl value "net.netfilter.nf_conntrack_max" maxPerCore: 0 # Skip setting "net.netfilter.nf_conntrack_tcp_timeout_established" tcpEstablishedTimeout: 0s # Skip setting "net.netfilter.nf_conntrack_tcp_timeout_close" tcpCloseWaitTimeout: 0s CaveatsMost of "non-local" volume drivers such as nfs and iscsi do not work. Local volumes like local, hostPath, emptyDir, configMap, secret, and downwardAPI are known to work.Some CNI plugins may not work. Flannel (VXLAN) is known to work.For more on this, see the Caveats and Future work page on the rootlesscontaine.rs website.See Alsorootlesscontaine.rsRootless Containers 2020 (KubeCon NA 2020)Running kind with Rootless DockerUsernetesRunning K3s with rootless modeKEP-2033: Kubelet-in-UserNS (aka Rootless mode)Items on this page refer to third party products or projects that provide functionality required by Kubernetes. The Kubernetes project authors aren't responsible for those third-party products or projects. See the CNCF website guidelines for more details.You should read the content guide before proposing a change that adds an extra third-party link.