# kube-apiserver Configuration (v1alpha1)

kube-apiserver Configuration (v1alpha1)Package v1alpha1 is the v1alpha1 version of the API.Resource TypesAdmissionConfigurationAuthenticationConfigurationAuthorizationConfigurationEgressSelectorConfigurationTracingConfigurationTracingConfigurationAppears in:KubeletConfigurationTracingConfigurationTracingConfiguration provides versioned configuration for OpenTelemetry tracing clients.FieldDescriptionendpointstringEndpoint of the collector this component will report traces to. The connection is insecure, and does not currently support TLS. Recommended is unset, and endpoint is the otlp grpc default, localhost:4317.samplingRatePerMillionint32SamplingRatePerMillion is the number of samples to collect per million spans. Recommended is unset. If unset, sampler respects its parent span's sampling rate, but otherwise never samples.AdmissionConfigurationAdmissionConfiguration provides versioned configuration for admission controllers.FieldDescriptionapiVersionstringapiserver.k8s.io/v1alpha1kindstringAdmissionConfigurationplugins[]AdmissionPluginConfigurationPlugins allows specifying a configuration per admission control plugin.AuthenticationConfigurationAuthenticationConfiguration provides versioned configuration for authentication.FieldDescriptionapiVersionstringapiserver.k8s.io/v1alpha1kindstringAuthenticationConfigurationjwt [Required][]JWTAuthenticatorjwt is a list of authenticator to authenticate Kubernetes users using JWT compliant tokens. The authenticator will attempt to parse a raw ID token, verify it's been signed by the configured issuer. The public key to verify the signature is discovered from the issuer's public endpoint using OIDC discovery. For an incoming token, each JWT authenticator will be attempted in the order in which it is specified in this list. Note however that other authenticators may run before or after the JWT authenticators. The specific position of JWT authenticators in relation to other authenticators is neither defined nor stable across releases. Since each JWT authenticator must have a unique issuer URL, at most one JWT authenticator will attempt to cryptographically validate the token.The minimum valid JWT payload must contain the following claims: { "iss": "https://issuer.example.com", "aud": ["audience"], "exp": 1234567890, "": "username" }anonymous [Required]AnonymousAuthConfigIf present --anonymous-auth must not be setAuthorizationConfigurationFieldDescriptionapiVersionstringapiserver.k8s.io/v1alpha1kindstringAuthorizationConfigurationauthorizers [Required][]AuthorizerConfigurationAuthorizers is an ordered list of authorizers to authorize requests against. This is similar to the --authorization-modes kube-apiserver flag Must be at least one.EgressSelectorConfigurationEgressSelectorConfiguration provides versioned configuration for egress selector clients.FieldDescriptionapiVersionstringapiserver.k8s.io/v1alpha1kindstringEgressSelectorConfigurationegressSelections [Required][]EgressSelectionconnectionServices contains a list of egress selection client configurationsTracingConfigurationTracingConfiguration provides versioned configuration for tracing clients.FieldDescriptionapiVersionstringapiserver.k8s.io/v1alpha1kindstringTracingConfigurationTracingConfiguration [Required]TracingConfiguration(Members of TracingConfiguration are embedded into this type.)Embed the component config tracing configuration structAdmissionPluginConfigurationAppears in:AdmissionConfigurationAdmissionPluginConfiguration provides the configuration for a single plug-in.FieldDescriptionname [Required]stringName is the name of the admission controller. It must match the registered admission plugin name.pathstringPath is the path to a configuration file that contains the plugin's configurationconfigurationk8s.io/apimachinery/pkg/runtime.UnknownConfiguration is an embedded configuration object to be used as the plugin's configuration. If present, it will be used instead of the path to the configuration file.AnonymousAuthConditionAppears in:AnonymousAuthConfigAnonymousAuthCondition describes the condition under which anonymous auth should be enabled.FieldDescriptionpath [Required]stringPath for which anonymous auth is enabled.AnonymousAuthConfigAppears in:AuthenticationConfigurationAnonymousAuthConfig provides the configuration for the anonymous authenticator.FieldDescriptionenabled [Required]boolNo description provided.conditions [Required][]AnonymousAuthConditionIf set, anonymous auth is only allowed if the request meets one of the conditions.AudienceMatchPolicyType(Alias of string)Appears in:IssuerAudienceMatchPolicyType is a set of valid values for issuer.audienceMatchPolicyAuthorizerConfigurationAppears in:AuthorizationConfigurationFieldDescriptiontype [Required]stringType refers to the type of the authorizer "Webhook" is supported in the generic API server Other API servers may support additional authorizer types like Node, RBAC, ABAC, etc.name [Required]stringName used to describe the webhook This is explicitly used in monitoring machinery for metrics Note: Names must be DNS1123 labels like myauthorizername or subdomains like myauthorizer.example.domain Required, with no defaultwebhook [Required]WebhookConfigurationWebhook defines the configuration for a Webhook authorizer Must be defined when Type=Webhook Must not be defined when Type!=WebhookClaimMappingsAppears in:JWTAuthenticatorClaimMappings provides the configuration for claim mappingFieldDescriptionusername [Required]PrefixedClaimOrExpressionusername represents an option for the username attribute. The claim's value must be a singular string. Same as the --oidc-username-claim and --oidc-username-prefix flags. If username.expression is set, the expression must produce a string value. If username.expression uses 'claims.email', then 'claims.email_verified' must be used in username.expression or extra[*].valueExpression or claimValidationRules[<*>].expression. An example claim validation rule expression that matches the validation automatically applied when username.claim is set to 'email' is 'claims.?email_verified.orValue(true) == true'. By explicitly comparing the value to true, we let type-checking see the result will be a boolean, and to make sure a non-boolean email_verified claim will be caught at runtime.In the flag based approach, the --oidc-username-claim and --oidc-username-prefix are optional. If --oidc-username-claim is not set, the default value is "sub". For the authentication config, there is no defaulting for claim or prefix. The claim and prefix must be set explicitly. For claim, if --oidc-username-claim was not set with legacy flag approach, configure username.claim="sub" in the authentication config. For prefix: (1) --oidc-username-prefix="-", no prefix was added to the username. For the same behavior using authentication config, set username.prefix="" (2) --oidc-username-prefix="" and --oidc-username-claim != "email", prefix was "<value of --oidc-issuer-url>#". For the same behavior using authentication config, set username.prefix="#" (3) --oidc-username-prefix="". For the same behavior using authentication config, set username.prefix=""groupsPrefixedClaimOrExpressiongroups represents an option for the groups attribute. The claim's value must be a string or string array claim. If groups.claim is set, the prefix must be specified (and can be the empty string). If groups.expression is set, the expression must produce a string or string array value. "", [], and null values are treated as the group mapping not being present.uidClaimOrExpressionuid represents an option for the uid attribute. Claim must be a singular string claim. If uid.expression is set, the expression must produce a string value.extra[]ExtraMappingextra represents an option for the extra attribute. expression must produce a string or string array value. If the value is empty, the extra mapping will not be present.hard-coded extra key/valuekey: "foo" valueExpression: "'bar'" This will result in an extra attribute - foo: ["bar"]hard-coded key, value copying claim valuekey: "foo" valueExpression: "claims.some_claim" This will result in an extra attribute - foo: [value of some_claim]hard-coded key, value derived from claim valuekey: "admin" valueExpression: '(has(claims.is_admin) && claims.is_admin) ? "true":""' This will result in:if is_admin claim is present and true, extra attribute - admin: ["true"]if is_admin claim is present and false or is_admin claim is not present, no extra attribute will be addedClaimOrExpressionAppears in:ClaimMappingsClaimOrExpression provides the configuration for a single claim or expression.FieldDescriptionclaimstringclaim is the JWT claim to use. Either claim or expression must be set. Mutually exclusive with expression.expressionstringexpression represents the expression which will be evaluated by CEL.CEL expressions have access to the contents of the token claims, organized into CEL variable:'claims' is a map of claim names to claim values. For example, a variable named 'sub' can be accessed as 'claims.sub'. Nested claims can be accessed using dot notation, e.g. 'claims.foo.bar'.Documentation on CEL: https://kubernetes.io/docs/reference/using-api/cel/Mutually exclusive with claim.ClaimValidationRuleAppears in:JWTAuthenticatorClaimValidationRule provides the configuration for a single claim validation rule.FieldDescriptionclaimstringclaim is the name of a required claim. Same as --oidc-required-claim flag. Only string claim keys are supported. Mutually exclusive with expression and message.requiredValuestringrequiredValue is the value of a required claim. Same as --oidc-required-claim flag. Only string claim values are supported. If claim is set and requiredValue is not set, the claim must be present with a value set to the empty string. Mutually exclusive with expression and message.expressionstringexpression represents the expression which will be evaluated by CEL. Must produce a boolean.CEL expressions have access to the contents of the token claims, organized into CEL variable:'claims' is a map of claim names to claim values. For example, a variable named 'sub' can be accessed as 'claims.sub'. Nested claims can be accessed using dot notation, e.g. 'claims.foo.bar'. Must return true for the validation to pass.Documentation on CEL: https://kubernetes.io/docs/reference/using-api/cel/Mutually exclusive with claim and requiredValue.messagestringmessage customizes the returned error message when expression returns false. message is a literal string. Mutually exclusive with claim and requiredValue.ConnectionAppears in:EgressSelectionConnection provides the configuration for a single egress selection client.FieldDescriptionproxyProtocol [Required]ProtocolTypeProtocol is the protocol used to connect from client to the konnectivity server.transportTransportTransport defines the transport configurations we use to dial to the konnectivity server. This is required if ProxyProtocol is HTTPConnect or GRPC.EgressSelectionAppears in:EgressSelectorConfigurationEgressSelection provides the configuration for a single egress selection client.FieldDescriptionname [Required]stringname is the name of the egress selection. Currently supported values are "controlplane", "master", "etcd" and "cluster" The "master" egress selector is deprecated in favor of "controlplane"connection [Required]Connectionconnection is the exact information used to configure the egress selectionExtraMappingAppears in:ClaimMappingsExtraMapping provides the configuration for a single extra mapping.FieldDescriptionkey [Required]stringkey is a string to use as the extra attribute key. key must be a domain-prefix path (e.g. example.org/foo). All characters before the first "/" must be a valid subdomain as defined by RFC 1123. All characters trailing the first "/" must be valid HTTP Path characters as defined by RFC 3986. key must be lowercase. Required to be unique.valueExpression [Required]stringvalueExpression is a CEL expression to extract extra attribute value. valueExpression must produce a string or string array value. "", [], and null values are treated as the extra mapping not being present. Empty string values contained within a string array are filtered out.CEL expressions have access to the contents of the token claims, organized into CEL variable:'claims' is a map of claim names to claim values. For example, a variable named 'sub' can be accessed as 'claims.sub'. Nested claims can be accessed using dot notation, e.g. 'claims.foo.bar'.Documentation on CEL: https://kubernetes.io/docs/reference/using-api/cel/IssuerAppears in:JWTAuthenticatorIssuer provides the configuration for an external provider's specific settings.FieldDescriptionurl [Required]stringurl points to the issuer URL in a format https://url or https://url/path. This must match the "iss" claim in the presented JWT, and the issuer returned from discovery. Same value as the --oidc-issuer-url flag. Discovery information is fetched from "{url}/.well-known/openid-configuration" unless overridden by discoveryURL. Required to be unique across all JWT authenticators. Note that egress selection configuration is not used for this network connection.discoveryURLstringdiscoveryURL, if specified, overrides the URL used to fetch discovery information instead of using "{url}/.well-known/openid-configuration". The exact value specified is used, so "/.well-known/openid-configuration" must be included in discoveryURL if needed.The "issuer" field in the fetched discovery information must match the "issuer.url" field in the AuthenticationConfiguration and will be used to validate the "iss" claim in the presented JWT. This is for scenarios where the well-known and jwks endpoints are hosted at a different location than the issuer (such as locally in the cluster).Example: A discovery url that is exposed using kubernetes service 'oidc' in namespace 'oidc-namespace' and discovery information is available at '/.well-known/openid-configuration'. discoveryURL: "https://oidc.oidc-namespace/.well-known/openid-configuration" certificateAuthority is used to verify the TLS connection and the hostname on the leaf certificate must be set to 'oidc.oidc-namespace'.curl https://oidc.oidc-namespace/.well-known/openid-configuration (.discoveryURL field) { issuer: "https://oidc.example.com" (.url field) }discoveryURL must be different from url. Required to be unique across all JWT authenticators. Note that egress selection configuration is not used for this network connection.certificateAuthoritystringcertificateAuthority contains PEM-encoded certificate authority certificates used to validate the connection when fetching discovery information. If unset, the system verifier is used. Same value as the content of the file referenced by the --oidc-ca-file flag.audiences [Required][]stringaudiences is the set of acceptable audiences the JWT must be issued to. At least one of the entries must match the "aud" claim in presented JWTs. Same value as the --oidc-client-id flag (though this field supports an array). Required to be non-empty.audienceMatchPolicyAudienceMatchPolicyTypeaudienceMatchPolicy defines how the "audiences" field is used to match the "aud" claim in the presented JWT. Allowed values are:"MatchAny" when multiple audiences are specified andempty (or unset) or "MatchAny" when a single audience is specified.MatchAny: the "aud" claim in the presented JWT must match at least one of the entries in the "audiences" field. For example, if "audiences" is ["foo", "bar"], the "aud" claim in the presented JWT must contain either "foo" or "bar" (and may contain both)."": The match policy can be empty (or unset) when a single audience is specified in the "audiences" field. The "aud" claim in the presented JWT must contain the single audience (and may contain others).For more nuanced audience validation, use claimValidationRules. example: claimValidationRule[].expression: 'sets.equivalent(claims.aud, ["bar", "foo", "baz"])' to require an exact match.JWTAuthenticatorAppears in:AuthenticationConfigurationJWTAuthenticator provides the configuration for a single JWT authenticator.FieldDescriptionissuer [Required]Issuerissuer contains the basic OIDC provider connection options.claimValidationRules[]ClaimValidationRuleclaimValidationRules are rules that are applied to validate token claims to authenticate users.claimMappings [Required]ClaimMappingsclaimMappings points claims of a token to be treated as user attributes.userValidationRules[]UserValidationRuleuserValidationRules are rules that are applied to final user before completing authentication. These allow invariants to be applied to incoming identities such as preventing the use of the system: prefix that is commonly used by Kubernetes components. The validation rules are logically ANDed together and must all return true for the validation to pass.PrefixedClaimOrExpressionAppears in:ClaimMappingsPrefixedClaimOrExpression provides the configuration for a single prefixed claim or expression.FieldDescriptionclaimstringclaim is the JWT claim to use. Mutually exclusive with expression.prefixstringprefix is prepended to claim's value to prevent clashes with existing names. prefix needs to be set if claim is set and can be the empty string. Mutually exclusive with expression.expressionstringexpression represents the expression which will be evaluated by CEL.CEL expressions have access to the contents of the token claims, organized into CEL variable:'claims' is a map of claim names to claim values. For example, a variable named 'sub' can be accessed as 'claims.sub'. Nested claims can be accessed using dot notation, e.g. 'claims.foo.bar'.Documentation on CEL: https://kubernetes.io/docs/reference/using-api/cel/Mutually exclusive with claim and prefix.ProtocolType(Alias of string)Appears in:ConnectionProtocolType is a set of valid values for Connection.ProtocolTypeTCPTransportAppears in:TransportTCPTransport provides the information to connect to konnectivity server via TCPFieldDescriptionurl [Required]stringURL is the location of the konnectivity server to connect to. As an example it might be "https://127.0.0.1:8131"tlsConfigTLSConfigTLSConfig is the config needed to use TLS when connecting to konnectivity serverTLSConfigAppears in:TCPTransportTLSConfig provides the authentication information to connect to konnectivity server Only used with TCPTransportFieldDescriptioncaBundlestringcaBundle is the file location of the CA to be used to determine trust with the konnectivity server. Must be absent/empty if TCPTransport.URL is prefixed with http:// If absent while TCPTransport.URL is prefixed with https://, default to system trust roots.clientKeystringclientKey is the file location of the client key to be used in mtls handshakes with the konnectivity server. Must be absent/empty if TCPTransport.URL is prefixed with http:// Must be configured if TCPTransport.URL is prefixed with https://clientCertstringclientCert is the file location of the client certificate to be used in mtls handshakes with the konnectivity server. Must be absent/empty if TCPTransport.URL is prefixed with http:// Must be configured if TCPTransport.URL is prefixed with https://TransportAppears in:ConnectionTransport defines the transport configurations we use to dial to the konnectivity serverFieldDescriptiontcpTCPTransportTCP is the TCP configuration for communicating with the konnectivity server via TCP ProxyProtocol of GRPC is not supported with TCP transport at the moment Requires at least one of TCP or UDS to be setudsUDSTransportUDS is the UDS configuration for communicating with the konnectivity server via UDS Requires at least one of TCP or UDS to be setUDSTransportAppears in:TransportUDSTransport provides the information to connect to konnectivity server via UDSFieldDescriptionudsName [Required]stringUDSName is the name of the unix domain socket to connect to konnectivity server This does not use a unix:// prefix. (Eg: /etc/srv/kubernetes/konnectivity-server/konnectivity-server.socket)UserValidationRuleAppears in:JWTAuthenticatorUserValidationRule provides the configuration for a single user info validation rule.FieldDescriptionexpression [Required]stringexpression represents the expression which will be evaluated by CEL. Must return true for the validation to pass.CEL expressions have access to the contents of UserInfo, organized into CEL variable:'user' - authentication.k8s.io/v1, Kind=UserInfo object Refer to https://github.com/kubernetes/api/blob/release-1.28/authentication/v1/types.go#L105-L122 for the definition. API documentation: https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.28/#userinfo-v1-authentication-k8s-ioDocumentation on CEL: https://kubernetes.io/docs/reference/using-api/cel/messagestringmessage customizes the returned error message when rule returns false. message is a literal string.WebhookConfigurationAppears in:AuthorizerConfigurationFieldDescriptionauthorizedTTL [Required]meta/v1.DurationThe duration to cache 'authorized' responses from the webhook authorizer. Same as setting --authorization-webhook-cache-authorized-ttl flag Default: 5m0sunauthorizedTTL [Required]meta/v1.DurationThe duration to cache 'unauthorized' responses from the webhook authorizer. Same as setting --authorization-webhook-cache-unauthorized-ttl flag Default: 30stimeout [Required]meta/v1.DurationTimeout for the webhook request Maximum allowed value is 30s. Required, no default value.subjectAccessReviewVersion [Required]stringThe API version of the authorization.k8s.io SubjectAccessReview to send to and expect from the webhook. Same as setting --authorization-webhook-version flag Valid values: v1beta1, v1 Required, no default valuematchConditionSubjectAccessReviewVersion [Required]stringMatchConditionSubjectAccessReviewVersion specifies the SubjectAccessReview version the CEL expressions are evaluated against Valid values: v1 Required, no default valuefailurePolicy [Required]stringControls the authorization decision when a webhook request fails to complete or returns a malformed response or errors evaluating matchConditions. Valid values:NoOpinion: continue to subsequent authorizers to see if one of them allows the requestDeny: reject the request without consulting subsequent authorizers Required, with no default.connectionInfo [Required]WebhookConnectionInfoConnectionInfo defines how we talk to the webhookmatchConditions [Required][]WebhookMatchConditionmatchConditions is a list of conditions that must be met for a request to be sent to this webhook. An empty list of matchConditions matches all requests. There are a maximum of 64 match conditions allowed.The exact matching logic is (in order):If at least one matchCondition evaluates to FALSE, then the webhook is skipped.If ALL matchConditions evaluate to TRUE, then the webhook is called.If at least one matchCondition evaluates to an error (but none are FALSE):If failurePolicy=Deny, then the webhook rejects the requestIf failurePolicy=NoOpinion, then the error is ignored and the webhook is skippedWebhookConnectionInfoAppears in:WebhookConfigurationFieldDescriptiontype [Required]stringControls how the webhook should communicate with the server. Valid values:KubeConfigFile: use the file specified in kubeConfigFile to locate the server.InClusterConfig: use the in-cluster configuration to call the SubjectAccessReview API hosted by kube-apiserver. This mode is not allowed for kube-apiserver.kubeConfigFile [Required]stringPath to KubeConfigFile for connection info Required, if connectionInfo.Type is KubeConfigWebhookMatchConditionAppears in:WebhookConfigurationFieldDescriptionexpression [Required]stringexpression represents the expression which will be evaluated by CEL. Must evaluate to bool. CEL expressions have access to the contents of the SubjectAccessReview in v1 version. If version specified by subjectAccessReviewVersion in the request variable is v1beta1, the contents would be converted to the v1 version before evaluating the CEL expression.'resourceAttributes' describes information for a resource access request and is unset for non-resource requests. e.g. has(request.resourceAttributes) && request.resourceAttributes.namespace == 'default''nonResourceAttributes' describes information for a non-resource access request and is unset for resource requests. e.g. has(request.nonResourceAttributes) && request.nonResourceAttributes.path == '/healthz'.'user' is the user to test for. e.g. request.user == 'alice''groups' is the groups to test for. e.g. ('group1' in request.groups)'extra' corresponds to the user.Info.GetExtra() method from the authenticator.'uid' is the information about the requesting user. e.g. request.uid == '1'Documentation on CEL: https://kubernetes.io/docs/reference/using-api/cel/This page is automatically generated.If you plan to report an issue with this page, mention that the page is auto-generated in your issue description. The fix may need to happen elsewhere in the Kubernetes project.


# kubectl options

kubectl optionsSynopsisPrint the list of flags inherited by all commandskubectl options [flags] Examples # Print flags inherited by all commands kubectl options Options-h, --helphelp for optionsParent Options Inherited--as stringUsername to impersonate for the operation. User could be a regular user or a service account in a namespace.--as-group stringsGroup to impersonate for the operation, this flag can be repeated to specify multiple groups.--as-uid stringUID to impersonate for the operation.--cache-dir string Default: "$HOME/.kube/cache"Default cache directory--certificate-authority stringPath to a cert file for the certificate authority--client-certificate stringPath to a client certificate file for TLS--client-key stringPath to a client key file for TLS--cluster stringThe name of the kubeconfig cluster to use--context stringThe name of the kubeconfig context to use--disable-compressionIf true, opt-out of response compression for all requests to the server--insecure-skip-tls-verifyIf true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure--kubeconfig stringPath to the kubeconfig file to use for CLI requests.--kuberc stringPath to the kuberc file to use for preferences. This can be disabled by exporting KUBECTL_KUBERC=false feature gate or turning off the feature KUBERC=off.--match-server-versionRequire server version to match client version-n, --namespace stringIf present, the namespace scope for this CLI request--password stringPassword for basic authentication to the API server--profile string Default: "none"Name of profile to capture. One of (none|cpu|heap|goroutine|threadcreate|block|mutex)--profile-output string Default: "profile.pprof"Name of the file to write the profile to--request-timeout string Default: "0"The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests.-s, --server stringThe address and port of the Kubernetes API server--storage-driver-buffer-duration duration Default: 1m0sWrites in the storage driver will be buffered for this duration, and committed to the non memory backends as a single transaction--storage-driver-db string Default: "cadvisor"database name--storage-driver-host string Default: "localhost:8086"database host:port--storage-driver-password string Default: "root"database password--storage-driver-secureuse secure connection with database--storage-driver-table string Default: "stats"table name--storage-driver-user string Default: "root"database username--tls-server-name stringServer name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used--token stringBearer token for authentication to the API server--user stringThe name of the kubeconfig user to use--username stringUsername for basic authentication to the API server--version version[=true]--version, --version=raw prints version information and quits; --version=vX.Y.Z... sets the reported version--warnings-as-errorsTreat warnings received from the server as errors and exit with a non-zero exit codeSee Alsokubectl - kubectl controls the Kubernetes cluster managerThis page is automatically generated.If you plan to report an issue with this page, mention that the page is auto-generated in your issue description. The fix may need to happen elsewhere in the Kubernetes project.


# Install and Set Up kubectl on macOS

Install and Set Up kubectl on macOSBefore you beginYou must use a kubectl version that is within one minor version difference of your cluster. For example, a v1.34 client can communicate with v1.33, v1.34, and v1.35 control planes. Using the latest compatible version of kubectl helps avoid unforeseen issues.Install kubectl on macOSThe following methods exist for installing kubectl on macOS:Install kubectl on macOSInstall kubectl binary with curl on macOSInstall with Homebrew on macOSInstall with Macports on macOSVerify kubectl configurationOptional kubectl configurations and pluginsEnable shell autocompletionInstall kubectl convert pluginInstall kubectl binary with curl on macOSDownload the latest release: curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/darwin/amd64/kubectl" curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/darwin/arm64/kubectl" Note:To download a specific version, replace the $(curl -L -s https://dl.k8s.io/release/stable.txt) portion of the command with the specific version.For example, to download version 1.34.0 on Intel macOS, type:curl -LO "https://dl.k8s.io/release/v1.34.0/bin/darwin/amd64/kubectl" And for macOS on Apple Silicon, type:curl -LO "https://dl.k8s.io/release/v1.34.0/bin/darwin/arm64/kubectl" Validate the binary (optional)Download the kubectl checksum file: curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/darwin/amd64/kubectl.sha256" curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/darwin/arm64/kubectl.sha256" Validate the kubectl binary against the checksum file:echo "$(cat kubectl.sha256) kubectl" | shasum -a 256 --check If valid, the output is:kubectl: OK If the check fails, shasum exits with nonzero status and prints output similar to:kubectl: FAILED shasum: WARNING: 1 computed checksum did NOT match Note:Download the same version of the binary and checksum.Make the kubectl binary executable.chmod +x ./kubectl Move the kubectl binary to a file location on your system PATH.sudo mv ./kubectl /usr/local/bin/kubectl sudo chown root: /usr/local/bin/kubectl Note:Make sure /usr/local/bin is in your PATH environment variable.Test to ensure the version you installed is up-to-date:kubectl version --client Or use this for detailed view of version:kubectl version --client --output=yaml After installing and validating kubectl, delete the checksum file:rm kubectl.sha256 Install with Homebrew on macOSIf you are on macOS and using Homebrew package manager, you can install kubectl with Homebrew.Run the installation command:brew install kubectl orbrew install kubernetes-cli Test to ensure the version you installed is up-to-date:kubectl version --client Install with Macports on macOSIf you are on macOS and using Macports package manager, you can install kubectl with Macports.Run the installation command:sudo port selfupdate sudo port install kubectl Test to ensure the version you installed is up-to-date:kubectl version --client Verify kubectl configurationIn order for kubectl to find and access a Kubernetes cluster, it needs a kubeconfig file, which is created automatically when you create a cluster using kube-up.sh or successfully deploy a Minikube cluster. By default, kubectl configuration is located at ~/.kube/config.Check that kubectl is properly configured by getting the cluster state:kubectl cluster-info If you see a URL response, kubectl is correctly configured to access your cluster.If you see a message similar to the following, kubectl is not configured correctly or is not able to connect to a Kubernetes cluster.The connection to the server <server-name:port> was refused - did you specify the right host or port? For example, if you are intending to run a Kubernetes cluster on your laptop (locally), you will need a tool like Minikube to be installed first and then re-run the commands stated above.If kubectl cluster-info returns the url response but you can't access your cluster, to check whether it is configured properly, use:kubectl cluster-info dump Troubleshooting the 'No Auth Provider Found' error messageIn Kubernetes 1.26, kubectl removed the built-in authentication for the following cloud providers' managed Kubernetes offerings. These providers have released kubectl plugins to provide the cloud-specific authentication. For instructions, refer to the following provider documentation:Azure AKS: kubelogin pluginGoogle Kubernetes Engine: gke-gcloud-auth-plugin(There could also be other reasons to see the same error message, unrelated to that change.)Optional kubectl configurations and pluginsEnable shell autocompletionkubectl provides autocompletion support for Bash, Zsh, Fish, and PowerShell which can save you a lot of typing.Below are the procedures to set up autocompletion for Bash, Fish, and Zsh.IntroductionThe kubectl completion script for Bash can be generated with kubectl completion bash. Sourcing this script in your shell enables kubectl completion.However, the kubectl completion script depends on bash-completion which you thus have to previously install.Warning:There are two versions of bash-completion, v1 and v2. V1 is for Bash 3.2 (which is the default on macOS), and v2 is for Bash 4.1+. The kubectl completion script doesn't work correctly with bash-completion v1 and Bash 3.2. It requires bash-completion v2 and Bash 4.1+. Thus, to be able to correctly use kubectl completion on macOS, you have to install and use Bash 4.1+ (instructions). The following instructions assume that you use Bash 4.1+ (that is, any Bash version of 4.1 or newer).Upgrade BashThe instructions here assume you use Bash 4.1+. You can check your Bash's version by running:echo $BASH_VERSION If it is too old, you can install/upgrade it using Homebrew:brew install bash Reload your shell and verify that the desired version is being used:echo $BASH_VERSION $SHELL Homebrew usually installs it at /usr/local/bin/bash.Install bash-completionNote:As mentioned, these instructions assume you use Bash 4.1+, which means you will install bash-completion v2 (in contrast to Bash 3.2 and bash-completion v1, in which case kubectl completion won't work).You can test if you have bash-completion v2 already installed with type _init_completion. If not, you can install it with Homebrew:brew install bash-completion@2 As stated in the output of this command, add the following to your ~/.bash_profile file:brew_etc="$(brew --prefix)/etc" && [[ -r "${brew_etc}/profile.d/bash_completion.sh" ]] && . "${brew_etc}/profile.d/bash_completion.sh" Reload your shell and verify that bash-completion v2 is correctly installed with type _init_completion.Enable kubectl autocompletionYou now have to ensure that the kubectl completion script gets sourced in all your shell sessions. There are multiple ways to achieve this:Source the completion script in your ~/.bash_profile file:echo 'source <(kubectl completion bash)' >>~/.bash_profile Add the completion script to the /usr/local/etc/bash_completion.d directory:kubectl completion bash >/usr/local/etc/bash_completion.d/kubectl If you have an alias for kubectl, you can extend shell completion to work with that alias:echo 'alias k=kubectl' >>~/.bash_profile echo 'complete -o default -F __start_kubectl k' >>~/.bash_profile If you installed kubectl with Homebrew (as explained here), then the kubectl completion script should already be in /usr/local/etc/bash_completion.d/kubectl. In that case, you don't need to do anything.Note:The Homebrew installation of bash-completion v2 sources all the files in the BASH_COMPLETION_COMPAT_DIR directory, that's why the latter two methods work.In any case, after reloading your shell, kubectl completion should be working.Note:Autocomplete for Fish requires kubectl 1.23 or later.The kubectl completion script for Fish can be generated with the command kubectl completion fish. Sourcing the completion script in your shell enables kubectl autocompletion.To do so in all your shell sessions, add the following line to your ~/.config/fish/config.fish file:kubectl completion fish | source After reloading your shell, kubectl autocompletion should be working.The kubectl completion script for Zsh can be generated with the command kubectl completion zsh. Sourcing the completion script in your shell enables kubectl autocompletion.To do so in all your shell sessions, add the following to your ~/.zshrc file:source <(kubectl completion zsh) If you have an alias for kubectl, kubectl autocompletion will automatically work with it.After reloading your shell, kubectl autocompletion should be working.If you get an error like 2: command not found: compdef, then add the following to the beginning of your ~/.zshrc file:autoload -Uz compinit compinit Configure kubercSee kuberc for more information.Install kubectl convert pluginA plugin for Kubernetes command-line tool kubectl, which allows you to convert manifests between different API versions. This can be particularly helpful to migrate manifests to a non-deprecated api version with newer Kubernetes release. For more info, visit migrate to non deprecated apisDownload the latest release with the command: curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/darwin/amd64/kubectl-convert" curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/darwin/arm64/kubectl-convert" Validate the binary (optional)Download the kubectl-convert checksum file: curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/darwin/amd64/kubectl-convert.sha256" curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/darwin/arm64/kubectl-convert.sha256" Validate the kubectl-convert binary against the checksum file:echo "$(cat kubectl-convert.sha256) kubectl-convert" | shasum -a 256 --check If valid, the output is:kubectl-convert: OK If the check fails, shasum exits with nonzero status and prints output similar to:kubectl-convert: FAILED shasum: WARNING: 1 computed checksum did NOT match Note:Download the same version of the binary and checksum.Make kubectl-convert binary executablechmod +x ./kubectl-convert Move the kubectl-convert binary to a file location on your system PATH.sudo mv ./kubectl-convert /usr/local/bin/kubectl-convert sudo chown root: /usr/local/bin/kubectl-convert Note:Make sure /usr/local/bin is in your PATH environment variable.Verify plugin is successfully installedkubectl convert --help If you do not see an error, it means the plugin is successfully installed.After installing the plugin, clean up the installation files:rm kubectl-convert kubectl-convert.sha256 Uninstall kubectl on macOSDepending on how you installed kubectl, use one of the following methods.Uninstall kubectl using the command-lineLocate the kubectl binary on your system:which kubectl Remove the kubectl binary:sudo rm <path> Replace <path> with the path to the kubectl binary from the previous step. For example, sudo rm /usr/local/bin/kubectl.Uninstall kubectl using homebrewIf you installed kubectl using Homebrew, run the following command:brew remove kubectl What's nextInstall MinikubeSee the getting started guides for more about creating clusters.Learn how to launch and expose your application.If you need access to a cluster you didn't create, see the Sharing Cluster Access document.Read the kubectl reference docs


# Role Based Access Control Good Practices

Role Based Access Control Good PracticesPrinciples and practices for good RBAC design for cluster operators.Kubernetes RBAC is a key security control to ensure that cluster users and workloads have only the access to resources required to execute their roles. It is important to ensure that, when designing permissions for cluster users, the cluster administrator understands the areas where privilege escalation could occur, to reduce the risk of excessive access leading to security incidents.The good practices laid out here should be read in conjunction with the general RBAC documentation.General good practiceLeast privilegeIdeally, minimal RBAC rights should be assigned to users and service accounts. Only permissions explicitly required for their operation should be used. While each cluster will be different, some general rules that can be applied are :Assign permissions at the namespace level where possible. Use RoleBindings as opposed to ClusterRoleBindings to give users rights only within a specific namespace.Avoid providing wildcard permissions when possible, especially to all resources. As Kubernetes is an extensible system, providing wildcard access gives rights not just to all object types that currently exist in the cluster, but also to all object types which are created in the future.Administrators should not use cluster-admin accounts except where specifically needed. Providing a low privileged account with impersonation rights can avoid accidental modification of cluster resources.Avoid adding users to the system:masters group. Any user who is a member of this group bypasses all RBAC rights checks and will always have unrestricted superuser access, which cannot be revoked by removing RoleBindings or ClusterRoleBindings. As an aside, if a cluster is using an authorization webhook, membership of this group also bypasses that webhook (requests from users who are members of that group are never sent to the webhook)Minimize distribution of privileged tokensIdeally, pods shouldn't be assigned service accounts that have been granted powerful permissions (for example, any of the rights listed under privilege escalation risks). In cases where a workload requires powerful permissions, consider the following practices:Limit the number of nodes running powerful pods. Ensure that any DaemonSets you run are necessary and are run with least privilege to limit the blast radius of container escapes.Avoid running powerful pods alongside untrusted or publicly-exposed ones. Consider using Taints and Toleration, NodeAffinity, or PodAntiAffinity to ensure pods don't run alongside untrusted or less-trusted Pods. Pay special attention to situations where less-trustworthy Pods are not meeting the Restricted Pod Security Standard.HardeningKubernetes defaults to providing access which may not be required in every cluster. Reviewing the RBAC rights provided by default can provide opportunities for security hardening. In general, changes should not be made to rights provided to system: accounts some options to harden cluster rights exist:Review bindings for the system:unauthenticated group and remove them where possible, as this gives access to anyone who can contact the API server at a network level.Avoid the default auto-mounting of service account tokens by setting automountServiceAccountToken: false. For more details, see using default service account token. Setting this value for a Pod will overwrite the service account setting, workloads which require service account tokens can still mount them.Periodic reviewIt is vital to periodically review the Kubernetes RBAC settings for redundant entries and possible privilege escalations. If an attacker is able to create a user account with the same name as a deleted user, they can automatically inherit all the rights of the deleted user, especially the rights assigned to that user.Kubernetes RBAC - privilege escalation risksWithin Kubernetes RBAC there are a number of privileges which, if granted, can allow a user or a service account to escalate their privileges in the cluster or affect systems outside the cluster.This section is intended to provide visibility of the areas where cluster operators should take care, to ensure that they do not inadvertently allow for more access to clusters than intended.Listing secretsIt is generally clear that allowing get access on Secrets will allow a user to read their contents. It is also important to note that list and watch access also effectively allow for users to reveal the Secret contents. For example, when a List response is returned (for example, via kubectl get secrets -A -o yaml), the response includes the contents of all Secrets.Workload creationPermission to create workloads (either Pods, or workload resources that manage Pods) in a namespace implicitly grants access to many other resources in that namespace, such as Secrets, ConfigMaps, and PersistentVolumes that can be mounted in Pods. Additionally, since Pods can run as any ServiceAccount, granting permission to create workloads also implicitly grants the API access levels of any service account in that namespace.Users who can run privileged Pods can use that access to gain node access and potentially to further elevate their privileges. Where you do not fully trust a user or other principal with the ability to create suitably secure and isolated Pods, you should enforce either the Baseline or Restricted Pod Security Standard. You can use Pod Security admission or other (third party) mechanisms to implement that enforcement.For these reasons, namespaces should be used to separate resources requiring different levels of trust or tenancy. It is still considered best practice to follow least privilege principles and assign the minimum set of permissions, but boundaries within a namespace should be considered weak.Persistent volume creationIf someone - or some application - is allowed to create arbitrary PersistentVolumes, that access includes the creation of hostPath volumes, which then means that a Pod would get access to the underlying host filesystem(s) on the associated node. Granting that ability is a security risk.There are many ways a container with unrestricted access to the host filesystem can escalate privileges, including reading data from other containers, and abusing the credentials of system services, such as Kubelet.You should only allow access to create PersistentVolume objects for:Users (cluster operators) that need this access for their work, and who you trust.The Kubernetes control plane components which creates PersistentVolumes based on PersistentVolumeClaims that are configured for automatic provisioning. This is usually setup by the Kubernetes provider or by the operator when installing a CSI driver.Where access to persistent storage is required trusted administrators should create PersistentVolumes, and constrained users should use PersistentVolumeClaims to access that storage.Access to proxy subresource of NodesUsers with access to the proxy sub-resource of node objects have rights to the Kubelet API, which allows for command execution on every pod on the node(s) to which they have rights. This access bypasses audit logging and admission control, so care should be taken before granting rights to this resource.Escalate verbGenerally, the RBAC system prevents users from creating clusterroles with more rights than the user possesses. The exception to this is the escalate verb. As noted in the RBAC documentation, users with this right can effectively escalate their privileges.Bind verbSimilar to the escalate verb, granting users this right allows for the bypass of Kubernetes in-built protections against privilege escalation, allowing users to create bindings to roles with rights they do not already have.Impersonate verbThis verb allows users to impersonate and gain the rights of other users in the cluster. Care should be taken when granting it, to ensure that excessive permissions cannot be gained via one of the impersonated accounts.CSRs and certificate issuingThe CSR API allows for users with create rights to CSRs and update rights on certificatesigningrequests/approval where the signer is kubernetes.io/kube-apiserver-client to create new client certificates which allow users to authenticate to the cluster. Those client certificates can have arbitrary names including duplicates of Kubernetes system components. This will effectively allow for privilege escalation.Token requestUsers with create rights on serviceaccounts/token can create TokenRequests to issue tokens for existing service accounts.Control admission webhooksUsers with control over validatingwebhookconfigurations or mutatingwebhookconfigurations can control webhooks that can read any object admitted to the cluster, and in the case of mutating webhooks, also mutate admitted objects.Namespace modificationUsers who can perform patch operations on Namespace objects (through a namespaced RoleBinding to a Role with that access) can modify labels on that namespace. In clusters where Pod Security Admission is used, this may allow a user to configure the namespace for a more permissive policy than intended by the administrators. For clusters where NetworkPolicy is used, users may be set labels that indirectly allow access to services that an administrator did not intend to allow.Kubernetes RBAC - denial of service risksObject creation denial-of-serviceUsers who have rights to create objects in a cluster may be able to create sufficient large objects to create a denial of service condition either based on the size or number of objects, as discussed in etcd used by Kubernetes is vulnerable to OOM attack. This may be specifically relevant in multi-tenant clusters if semi-trusted or untrusted users are allowed limited access to a system.One option for mitigation of this issue would be to use resource quotas to limit the quantity of objects which can be created.What's nextTo learn more about RBAC, see the RBAC documentation.


# Namespaces

NamespacesIn Kubernetes, namespaces provide a mechanism for isolating groups of resources within a single cluster. Names of resources need to be unique within a namespace, but not across namespaces. Namespace-based scoping is applicable only for namespaced objects (e.g. Deployments, Services, etc.) and not for cluster-wide objects (e.g. StorageClass, Nodes, PersistentVolumes, etc.).When to Use Multiple NamespacesNamespaces are intended for use in environments with many users spread across multiple teams, or projects. For clusters with a few to tens of users, you should not need to create or think about namespaces at all. Start using namespaces when you need the features they provide.Namespaces provide a scope for names. Names of resources need to be unique within a namespace, but not across namespaces. Namespaces cannot be nested inside one another and each Kubernetes resource can only be in one namespace.Namespaces are a way to divide cluster resources between multiple users (via resource quota).It is not necessary to use multiple namespaces to separate slightly different resources, such as different versions of the same software: use labels to distinguish resources within the same namespace.Note:For a production cluster, consider not using the default namespace. Instead, make other namespaces and use those.Initial namespacesKubernetes starts with four initial namespaces:defaultKubernetes includes this namespace so that you can start using your new cluster without first creating a namespace.kube-node-leaseThis namespace holds Lease objects associated with each node. Node leases allow the kubelet to send heartbeats so that the control plane can detect node failure.kube-publicThis namespace is readable by all clients (including those not authenticated). This namespace is mostly reserved for cluster usage, in case that some resources should be visible and readable publicly throughout the whole cluster. The public aspect of this namespace is only a convention, not a requirement.kube-systemThe namespace for objects created by the Kubernetes system.Working with NamespacesCreation and deletion of namespaces are described in the Admin Guide documentation for namespaces.Note:Avoid creating namespaces with the prefix kube-, since it is reserved for Kubernetes system namespaces.Viewing namespacesYou can list the current namespaces in a cluster using:kubectl get namespace NAME STATUS AGE default Active 1d kube-node-lease Active 1d kube-public Active 1d kube-system Active 1d Setting the namespace for a requestTo set the namespace for a current request, use the --namespace flag.For example:kubectl run nginx --image=nginx --namespace=<insert-namespace-name-here> kubectl get pods --namespace=<insert-namespace-name-here> Setting the namespace preferenceYou can permanently save the namespace for all subsequent kubectl commands in that context.kubectl config set-context --current --namespace=<insert-namespace-name-here> # Validate it kubectl config view --minify | grep namespace: Namespaces and DNSWhen you create a Service, it creates a corresponding DNS entry. This entry is of the form <service-name>.<namespace-name>.svc.cluster.local, which means that if a container only uses <service-name>, it will resolve to the service which is local to a namespace. This is useful for using the same configuration across multiple namespaces such as Development, Staging and Production. If you want to reach across namespaces, you need to use the fully qualified domain name (FQDN).As a result, all namespace names must be valid RFC 1123 DNS labels.Warning:By creating namespaces with the same name as public top-level domains, Services in these namespaces can have short DNS names that overlap with public DNS records. Workloads from any namespace performing a DNS lookup without a trailing dot will be redirected to those services, taking precedence over public DNS.To mitigate this, limit privileges for creating namespaces to trusted users. If required, you could additionally configure third-party security controls, such as admission webhooks, to block creating any namespace with the name of public TLDs.Not all objects are in a namespaceMost Kubernetes resources (e.g. pods, services, replication controllers, and others) are in some namespaces. However namespace resources are not themselves in a namespace. And low-level resources, such as nodes and persistentVolumes, are not in any namespace.To see which Kubernetes resources are and aren't in a namespace:# In a namespace kubectl api-resources --namespaced=true # Not in a namespace kubectl api-resources --namespaced=false Automatic labellingFEATURE STATE: Kubernetes 1.22 [stable]The Kubernetes control plane sets an immutable label kubernetes.io/metadata.name on all namespaces. The value of the label is the namespace name.What's nextLearn more about creating a new namespace.Learn more about deleting a namespace.


# Application Security Checklist

Application Security ChecklistBaseline guidelines around ensuring application security on Kubernetes, aimed at application developersThis checklist aims to provide basic guidelines on securing applications running in Kubernetes from a developer's perspective. This list is not meant to be exhaustive and is intended to evolve over time.On how to read and use this document:The order of topics does not reflect an order of priority.Some checklist items are detailed in the paragraph below the list of each section.This checklist assumes that a developer is a Kubernetes cluster user who interacts with namespaced scope objects.Caution:Checklists are not sufficient for attaining a good security posture on their own. A good security posture requires constant attention and improvement, but a checklist can be the first step on the never-ending journey towards security preparedness. Some recommendations in this checklist may be too restrictive or too lax for your specific security needs. Since Kubernetes security is not "one size fits all", each category of checklist items should be evaluated on its merits.Base security hardeningThe following checklist provides base security hardening recommendations that would apply to most applications deploying to Kubernetes.Application design Follow the right security principles when designing applications. Application configured with appropriate QoS class through resource request and limits. Memory limit is set for the workloads with a limit equal to or greater than the request. CPU limit might be set on sensitive workloads.Service account Avoid using the default ServiceAccount. Instead, create ServiceAccounts for each workload or microservice. automountServiceAccountToken should be set to false unless the pod specifically requires access to the Kubernetes API to operate.Pod-level securityContext recommendations Set runAsNonRoot: true. Configure the container to execute as a less privileged user (for example, using runAsUser and runAsGroup), and configure appropriate permissions on files or directories inside the container image. Optionally add a supplementary group with fsGroup to access persistent volumes. The application deploys into a namespace that enforces an appropriate Pod security standard. If you cannot control this enforcement for the cluster(s) where the application is deployed, take this into account either through documentation or additional defense in depth.Container-level securityContext recommendations Disable privilege escalations using allowPrivilegeEscalation: false. Configure the root filesystem to be read-only with readOnlyRootFilesystem: true. Avoid running privileged containers (set privileged: false). Drop all capabilities from the containers and add back only specific ones that are needed for operation of the container.Role Based Access Control (RBAC) Permissions such as create, patch, update and delete should be only granted if necessary. Avoid creating RBAC permissions to create or update roles which can lead to privilege escalation. Review bindings for the system:unauthenticated group and remove them where possible, as this gives access to anyone who can contact the API server at a network level.The create, update and delete verbs should be permitted judiciously. The patch verb if allowed on a Namespace can allow users to update labels on the namespace or deployments which can increase the attack surface.For sensitive workloads, consider providing a recommended ValidatingAdmissionPolicy that further restricts the permitted write actions.Image security Using an image scanning tool to scan an image before deploying containers in the Kubernetes cluster. Use container signing to validate the container image signature before deploying to the Kubernetes cluster.Network policies Configure NetworkPolicies to only allow expected ingress and egress traffic from the pods.Make sure that your cluster provides and enforces NetworkPolicy. If you are writing an application that users will deploy to different clusters, consider whether you can assume that NetworkPolicy is available and enforced.Advanced security hardeningThis section of this guide covers some advanced security hardening points which might be valuable based on different Kubernetes environment setup.Linux container securityConfigure Security Context for the pod-container. Set the Seccomp Profile for a Container. Restrict a Container's Access to Resources with AppArmor. Assign SELinux Labels to a Container.Runtime classes Configure appropriate runtime classes for containers.Note: This section links to third party projects that provide functionality required by Kubernetes. The Kubernetes project authors aren't responsible for these projects, which are listed alphabetically. To add a project to this list, read the content guide before submitting a change. More information.Some containers may require a different isolation level from what is provided by the default runtime of the cluster. runtimeClassName can be used in a podspec to define a different runtime class.For sensitive workloads consider using kernel emulation tools like gVisor, or virtualized isolation using a mechanism such as kata-containers.In high trust environments, consider using confidential virtual machines to improve cluster security even further.Items on this page refer to third party products or projects that provide functionality required by Kubernetes. The Kubernetes project authors aren't responsible for those third-party products or projects. See the CNCF website guidelines for more details.You should read the content guide before proposing a change that adds an extra third-party link.


# Monitoring, Logging, and Debugging

Monitoring, Logging, and DebuggingSet up monitoring and logging to troubleshoot a cluster, or debug a containerized application.Sometimes things go wrong. This guide helps you gather the relevant information and resolve issues. It has four sections:Debugging your application - Useful for users who are deploying code into Kubernetes and wondering why it is not working.Debugging your cluster - Useful for cluster administrators and operators troubleshooting issues with the Kubernetes cluster itself.Logging in Kubernetes - Useful for cluster administrators who want to set up and manage logging in Kubernetes.Monitoring in Kubernetes - Useful for cluster administrators who want to enable monitoring in a Kubernetes cluster.You should also check the known issues for the release you're using.Getting helpIf your problem isn't answered by any of the guides above, there are variety of ways for you to get help from the Kubernetes community.QuestionsThe documentation on this site has been structured to provide answers to a wide range of questions. Concepts explain the Kubernetes architecture and how each component works, while Setup provides practical instructions for getting started. Tasks show how to accomplish commonly used tasks, and Tutorials are more comprehensive walkthroughs of real-world, industry-specific, or end-to-end development scenarios. The Reference section provides detailed documentation on the Kubernetes API and command-line interfaces (CLIs), such as kubectl.Help! My question isn't covered! I need help now!Stack Exchange, Stack Overflow, or Server FaultIf you have questions related to software development for your containerized app, you can ask those on Stack Overflow.If you have Kubernetes questions related to cluster management or configuration, you can ask those on Server Fault.There are also several more specific Stack Exchange network sites which might be the right place to ask Kubernetes questions in areas such as DevOps, Software Engineering, or InfoSec.Someone else from the community may have already asked a similar question or may be able to help with your problem.The Kubernetes team will also monitor posts tagged Kubernetes. If there aren't any existing questions that help, please ensure that your question is on-topic on Stack Overflow, Server Fault, or the Stack Exchange Network site you're asking on, and read through the guidance on how to ask a new question, before asking a new one!SlackMany people from the Kubernetes community hang out on Kubernetes Slack in the #kubernetes-users channel. Slack requires registration; you can request an invitation, and registration is open to everyone). Feel free to come and ask any and all questions. Once registered, access the Kubernetes organisation in Slack via your web browser or via Slack's own dedicated app.Once you are registered, browse the growing list of channels for various subjects of interest. For example, people new to Kubernetes may also want to join the #kubernetes-novice channel. As another example, developers should join the #kubernetes-contributors channel.There are also many country specific / local language channels. Feel free to join these channels for localized support and info:Country / language specific Slack channelsCountryChannelsChina#cn-users, #cn-eventsFinland#fi-usersFrance#fr-users, #fr-eventsGermany#de-users, #de-eventsIndia#in-users, #in-eventsItaly#it-users, #it-eventsJapan#jp-users, #jp-eventsKorea#kr-usersNetherlands#nl-usersNorway#norw-usersPoland#pl-usersRussia#ru-usersSpain#es-usersSweden#se-usersTurkey#tr-users, #tr-eventsForumYou're welcome to join the official Kubernetes Forum: discuss.kubernetes.io.Bugs and feature requestsIf you have what looks like a bug, or you would like to make a feature request, please use the GitHub issue tracking system.Before you file an issue, please search existing issues to see if your issue is already covered.If filing a bug, please include detailed information about how to reproduce the problem, such as:Kubernetes version: kubectl versionCloud provider, OS distro, network configuration, and container runtime versionSteps to reproduce the problem


# Customizing DNS Service

Customizing DNS ServiceThis page explains how to configure your DNS Pod(s) and customize the DNS resolution process in your cluster.Before you beginYou need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. It is recommended to run this tutorial on a cluster with at least two nodes that are not acting as control plane hosts. If you do not already have a cluster, you can create one by using minikube or you can use one of these Kubernetes playgrounds:iximiuz LabsKillercodaKodeKloudPlay with KubernetesYour cluster must be running the CoreDNS add-on.Your Kubernetes server must be at or later than version v1.12.To check the version, enter kubectl version.IntroductionDNS is a built-in Kubernetes service launched automatically using the addon manager cluster add-on.Note:The CoreDNS Service is named kube-dns in the metadata.name field.The intent is to ensure greater interoperability with workloads that relied on the legacy kube-dns Service name to resolve addresses internal to the cluster. Using a Service named kube-dns abstracts away the implementation detail of which DNS provider is running behind that common name.If you are running CoreDNS as a Deployment, it will typically be exposed as a Kubernetes Service with a static IP address. The kubelet passes DNS resolver information to each container with the --cluster-dns=<dns-service-ip> flag.DNS names also need domains. You configure the local domain in the kubelet with the flag --cluster-domain=<default-local-domain>.The DNS server supports forward lookups (A and AAAA records), port lookups (SRV records), reverse IP address lookups (PTR records), and more. For more information, see DNS for Services and Pods.If a Pod's dnsPolicy is set to default, it inherits the name resolution configuration from the node that the Pod runs on. The Pod's DNS resolution should behave the same as the node. But see Known issues.If you don't want this, or if you want a different DNS config for pods, you can use the kubelet's --resolv-conf flag. Set this flag to "" to prevent Pods from inheriting DNS. Set it to a valid file path to specify a file other than /etc/resolv.conf for DNS inheritance.CoreDNSCoreDNS is a general-purpose authoritative DNS server that can serve as cluster DNS, complying with the DNS specifications.CoreDNS ConfigMap optionsCoreDNS is a DNS server that is modular and pluggable, with plugins adding new functionalities. The CoreDNS server can be configured by maintaining a Corefile, which is the CoreDNS configuration file. As a cluster administrator, you can modify the ConfigMap for the CoreDNS Corefile to change how DNS service discovery behaves for that cluster.In Kubernetes, CoreDNS is installed with the following default Corefile configuration:apiVersion: v1 kind: ConfigMap metadata: name: coredns namespace: kube-system data: Corefile: | .:53 { errors health { lameduck 5s } ready kubernetes cluster.local in-addr.arpa ip6.arpa { pods insecure fallthrough in-addr.arpa ip6.arpa ttl 30 } prometheus :9153 forward . /etc/resolv.conf cache 30 loop reload loadbalance } The Corefile configuration includes the following plugins of CoreDNS:errors: Errors are logged to stdout.health: Health of CoreDNS is reported to http://localhost:8080/health. In this extended syntax lameduck will make the process unhealthy then wait for 5 seconds before the process is shut down.ready: An HTTP endpoint on port 8181 will return 200 OK, when all plugins that are able to signal readiness have done so.kubernetes: CoreDNS will reply to DNS queries based on IP of the Services and Pods. You can find more details about this plugin on the CoreDNS website.ttl allows you to set a custom TTL for responses. The default is 5 seconds. The minimum TTL allowed is 0 seconds, and the maximum is capped at 3600 seconds. Setting TTL to 0 will prevent records from being cached.The pods insecure option is provided for backward compatibility with kube-dns.You can use the pods verified option, which returns an A record only if there exists a pod in the same namespace with a matching IP.The pods disabled option can be used if you don't use pod records.prometheus: Metrics of CoreDNS are available at http://localhost:9153/metrics in the Prometheus format (also known as OpenMetrics).forward: Any queries that are not within the Kubernetes cluster domain are forwarded to predefined resolvers (/etc/resolv.conf).cache: This enables a frontend cache.loop: Detects simple forwarding loops and halts the CoreDNS process if a loop is found.reload: Allows automatic reload of a changed Corefile. After you edit the ConfigMap configuration, allow two minutes for your changes to take effect.loadbalance: This is a round-robin DNS loadbalancer that randomizes the order of A, AAAA, and MX records in the answer.You can modify the default CoreDNS behavior by modifying the ConfigMap.Configuration of Stub-domain and upstream nameserver using CoreDNSCoreDNS has the ability to configure stub-domains and upstream nameservers using the forward plugin.ExampleIf a cluster operator has a Consul domain server located at "10.150.0.1", and all Consul names have the suffix ".consul.local". To configure it in CoreDNS, the cluster administrator creates the following stanza in the CoreDNS ConfigMap.consul.local:53 { errors cache 30 forward . 10.150.0.1 } To explicitly force all non-cluster DNS lookups to go through a specific nameserver at 172.16.0.1, point the forward to the nameserver instead of /etc/resolv.confforward . 172.16.0.1 The final ConfigMap along with the default Corefile configuration looks like:apiVersion: v1 kind: ConfigMap metadata: name: coredns namespace: kube-system data: Corefile: | .:53 { errors health kubernetes cluster.local in-addr.arpa ip6.arpa { pods insecure fallthrough in-addr.arpa ip6.arpa } prometheus :9153 forward . 172.16.0.1 cache 30 loop reload loadbalance } consul.local:53 { errors cache 30 forward . 10.150.0.1 } Note:CoreDNS does not support FQDNs for stub-domains and nameservers (eg: "ns.foo.com"). During translation, all FQDN nameservers will be omitted from the CoreDNS config.What's nextRead Debugging DNS Resolution


# Debugging Kubernetes Nodes With Kubectl

Debugging Kubernetes Nodes With KubectlThis page shows how to debug a node running on the Kubernetes cluster using kubectl debug command.Before you beginYou need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. It is recommended to run this tutorial on a cluster with at least two nodes that are not acting as control plane hosts. If you do not already have a cluster, you can create one by using minikube or you can use one of these Kubernetes playgrounds:iximiuz LabsKillercodaKodeKloudPlay with KubernetesYour Kubernetes server must be at or later than version 1.2.To check the version, enter kubectl version.You need to have permission to create Pods and to assign those new Pods to arbitrary nodes. You also need to be authorized to create Pods that access filesystems from the host.Debugging a Node using kubectl debug nodeUse the kubectl debug node command to deploy a Pod to a Node that you want to troubleshoot. This command is helpful in scenarios where you can't access your Node by using an SSH connection. When the Pod is created, the Pod opens an interactive shell on the Node. To create an interactive shell on a Node named “mynode”, run:kubectl debug node/mynode -it --image=ubuntu Creating debugging pod node-debugger-mynode-pdx84 with container debugger on node mynode. If you don't see a command prompt, try pressing enter. root@mynode:/# The debug command helps to gather information and troubleshoot issues. Commands that you might use include ip, ifconfig, nc, ping, and ps and so on. You can also install other tools, such as mtr, tcpdump, and curl, from the respective package manager.Note:The debug commands may differ based on the image the debugging pod is using and these commands might need to be installed.The debugging Pod can access the root filesystem of the Node, mounted at /host in the Pod. If you run your kubelet in a filesystem namespace, the debugging Pod sees the root for that namespace, not for the entire node. For a typical Linux node, you can look at the following paths to find relevant logs:/host/var/log/kubelet.logLogs from the kubelet, responsible for running containers on the node./host/var/log/kube-proxy.logLogs from kube-proxy, which is responsible for directing traffic to Service endpoints./host/var/log/containerd.logLogs from the containerd process running on the node./host/var/log/syslogShows general messages and information regarding the system./host/var/log/kern.logShows kernel logs.When creating a debugging session on a Node, keep in mind that:kubectl debug automatically generates the name of the new pod, based on the name of the node.The root filesystem of the Node will be mounted at /host.Although the container runs in the host IPC, Network, and PID namespaces, the pod isn't privileged. This means that reading some process information might fail because access to that information is restricted to superusers. For example, chroot /host will fail. If you need a privileged pod, create it manually or use the --profile=sysadmin flag.By applying Debugging Profiles, you can set specific properties such as securityContext to a debugging Pod.Cleaning upWhen you finish using the debugging Pod, delete it:kubectl get pods NAME READY STATUS RESTARTS AGE node-debugger-mynode-pdx84 0/1 Completed 0 8m1s # Change the pod name accordingly kubectl delete pod node-debugger-mynode-pdx84 --now pod "node-debugger-mynode-pdx84" deleted Note:The kubectl debug node command won't work if the Node is down (disconnected from the network, or kubelet dies and won't restart, etc.). Check debugging a down/unreachable node in that case.


# Writing a new topic

Writing a new topicThis page shows how to create a new topic for the Kubernetes docs.Before you beginCreate a fork of the Kubernetes documentation repository as described in Open a PR.Choosing a page typeAs you prepare to write a new topic, think about the page type that would fit your content the best:Guidelines for choosing a page typeTypeDescriptionConceptA concept page explains some aspect of Kubernetes. For example, a concept page might describe the Kubernetes Deployment object and explain the role it plays as an application while it is deployed, scaled, and updated. Typically, concept pages don't include sequences of steps, but instead provide links to tasks or tutorials. For an example of a concept topic, see Nodes.TaskA task page shows how to do a single thing. The idea is to give readers a sequence of steps that they can actually do as they read the page. A task page can be short or long, provided it stays focused on one area. In a task page, it is OK to blend brief explanations with the steps to be performed, but if you need to provide a lengthy explanation, you should do that in a concept topic. Related task and concept topics should link to each other. For an example of a short task page, see Configure a Pod to Use a Volume for Storage. For an example of a longer task page, see Configure Liveness and Readiness ProbesTutorialA tutorial page shows how to accomplish a goal that ties together several Kubernetes features. A tutorial might provide several sequences of steps that readers can actually do as they read the page. Or it might provide explanations of related pieces of code. For example, a tutorial could provide a walkthrough of a code sample. A tutorial can include brief explanations of the Kubernetes features that are being tied together, but should link to related concept topics for deep explanations of individual features.Creating a new pageUse a content type for each new page that you write. The docs site provides templates or Hugo archetypes to create new content pages. To create a new type of page, run hugo new with the path to the file you want to create. For example:hugo new docs/concepts/my-first-concept.md Choosing a title and filenameChoose a title that has the keywords you want search engines to find. Create a filename that uses the words in your title separated by hyphens. For example, the topic with title Using an HTTP Proxy to Access the Kubernetes API has filename http-proxy-access-api.md. You don't need to put "kubernetes" in the filename, because "kubernetes" is already in the URL for the topic, for example: /docs/tasks/extend-kubernetes/http-proxy-access-api/ Adding the topic title to the front matterIn your topic, put a title field in the front matter. The front matter is the YAML block that is between the triple-dashed lines at the top of the page. Here's an example:--- title: Using an HTTP Proxy to Access the Kubernetes API --- Choosing a directoryDepending on your page type, put your new file in a subdirectory of one of these:/content/en/docs/tasks//content/en/docs/tutorials//content/en/docs/concepts/You can put your file in an existing subdirectory, or you can create a new subdirectory.Placing your topic in the table of contentsThe table of contents is built dynamically using the directory structure of the documentation source. The top-level directories under /content/en/docs/ create top-level navigation, and subdirectories each have entries in the table of contents.Each subdirectory has a file _index.md, which represents the "home" page for a given subdirectory's content. The _index.md does not need a template. It can contain overview content about the topics in the subdirectory.Other files in a directory are sorted alphabetically by default. This is almost never the best order. To control the relative sorting of topics in a subdirectory, set the weight: front-matter key to an integer. Typically, we use multiples of 10, to account for adding topics later. For instance, a topic with weight 10 will come before one with weight 20.Embedding code in your topicIf you want to include some code in your topic, you can embed the code in your file directly using the markdown code block syntax. This is recommended for the following cases (not an exhaustive list):The code shows the output from a command such as kubectl get deploy mydeployment -o json | jq '.status'.The code is not generic enough for users to try out. As an example, you can embed the YAML file for creating a Pod which depends on a specific FlexVolume implementation.The code is an incomplete example because its purpose is to highlight a portion of a larger file. For example, when describing ways to customize a RoleBinding, you can provide a short snippet directly in your topic file.The code is not meant for users to try out due to other reasons. For example, when describing how a new attribute should be added to a resource using the kubectl edit command, you can provide a short example that includes only the attribute to add.Including code from another fileAnother way to include code in your topic is to create a new, complete sample file (or group of sample files) and then reference the sample from your topic. Use this method to include sample YAML files when the sample is generic and reusable, and you want the reader to try it out themselves.When adding a new standalone sample file, such as a YAML file, place the code in one of the <LANG>/examples/ subdirectories where <LANG> is the language for the topic. In your topic file, use the code_sample shortcode:{{% code_sample file="<RELPATH>/my-example-yaml>" %}} where <RELPATH> is the path to the file to include, relative to the examples directory. The following Hugo shortcode references a YAML file located at /content/en/examples/pods/storage/gce-volume.yaml.{{% code_sample file="pods/storage/gce-volume.yaml" %}} Showing how to create an API object from a configuration fileIf you need to demonstrate how to create an API object based on a configuration file, place the configuration file in one of the subdirectories under <LANG>/examples.In your topic, show this command:kubectl create -f https://k8s.io/examples/pods/storage/gce-volume.yaml Note:When adding new YAML files to the <LANG>/examples directory, make sure the file is also included into the <LANG>/examples_test.go file. The Travis CI for the Website automatically runs this test case when PRs are submitted to ensure all examples pass the tests.For an example of a topic that uses this technique, see Running a Single-Instance Stateful Application.Adding images to a topicPut image files in the /images directory. The preferred image format is SVG.What's nextLearn about using page content types.Learn about creating a pull request.


# Workloads

WorkloadsUnderstand Pods, the smallest deployable compute object in Kubernetes, and the higher-level abstractions that help you to run them.A workload is an application running on Kubernetes. Whether your workload is a single component or several that work together, on Kubernetes you run it inside a set of pods. In Kubernetes, a Pod represents a set of running containers on your cluster.Kubernetes pods have a defined lifecycle. For example, once a pod is running in your cluster then a critical fault on the node where that pod is running means that all the pods on that node fail. Kubernetes treats that level of failure as final: you would need to create a new Pod to recover, even if the node later becomes healthy.However, to make life considerably easier, you don't need to manage each Pod directly. Instead, you can use workload resources that manage a set of pods on your behalf. These resources configure controllers that make sure the right number of the right kind of pod are running, to match the state you specified.Kubernetes provides several built-in workload resources:Deployment and ReplicaSet (replacing the legacy resource ReplicationController). Deployment is a good fit for managing a stateless application workload on your cluster, where any Pod in the Deployment is interchangeable and can be replaced if needed.StatefulSet lets you run one or more related Pods that do track state somehow. For example, if your workload records data persistently, you can run a StatefulSet that matches each Pod with a PersistentVolume. Your code, running in the Pods for that StatefulSet, can replicate data to other Pods in the same StatefulSet to improve overall resilience.DaemonSet defines Pods that provide facilities that are local to nodes. Every time you add a node to your cluster that matches the specification in a DaemonSet, the control plane schedules a Pod for that DaemonSet onto the new node. Each pod in a DaemonSet performs a job similar to a system daemon on a classic Unix / POSIX server. A DaemonSet might be fundamental to the operation of your cluster, such as a plugin to run cluster networking, it might help you to manage the node, or it could provide optional behavior that enhances the container platform you are running.Job and CronJob provide different ways to define tasks that run to completion and then stop. You can use a Job to define a task that runs to completion, just once. You can use a CronJob to run the same Job multiple times according a schedule.In the wider Kubernetes ecosystem, you can find third-party workload resources that provide additional behaviors. Using a custom resource definition, you can add in a third-party workload resource if you want a specific behavior that's not part of Kubernetes' core. For example, if you wanted to run a group of Pods for your application but stop work unless all the Pods are available (perhaps for some high-throughput distributed task), then you can implement or install an extension that does provide that feature.What's nextAs well as reading about each API kind for workload management, you can read how to do specific tasks:Run a stateless application using a DeploymentRun a stateful application either as a single instance or as a replicated setRun automated tasks with a CronJobTo learn about Kubernetes' mechanisms for separating code from configuration, visit Configuration.There are two supporting concepts that provide backgrounds about how Kubernetes manages pods for applications:Garbage collection tidies up objects from your cluster after their owning resource has been removed.The time-to-live after finished controller removes Jobs once a defined time has passed since they completed.Once your application is running, you might want to make it available on the internet as a Service or, for web application only, using an Ingress.


# kubectl certificate deny

kubectl certificate denySynopsisDeny a certificate signing request.kubectl certificate deny allows a cluster admin to deny a certificate signing request (CSR). This action tells a certificate signing controller to not to issue a certificate to the requester.kubectl certificate deny (-f FILENAME | NAME) Examples # Deny CSR 'csr-sqgzp' kubectl certificate deny csr-sqgzp Options--allow-missing-template-keys Default: trueIf true, ignore any errors in templates when a field or map key is missing in the template. Only applies to golang and jsonpath output formats.-f, --filename stringsFilename, directory, or URL to files identifying the resource to update--forceUpdate the CSR even if it is already denied.-h, --helphelp for deny-k, --kustomize stringProcess the kustomization directory. This flag can't be used together with -f or -R.-o, --output stringOutput format. One of: (json, yaml, name, go-template, go-template-file, template, templatefile, jsonpath, jsonpath-as-json, jsonpath-file).-R, --recursiveProcess the directory used in -f, --filename recursively. Useful when you want to manage related manifests organized within the same directory.--show-managed-fieldsIf true, keep the managedFields when printing objects in JSON or YAML format.--template stringTemplate string or path to template file to use when -o=go-template, -o=go-template-file. The template format is golang templates [http://golang.org/pkg/text/template/#pkg-overview].Parent Options Inherited--as stringUsername to impersonate for the operation. User could be a regular user or a service account in a namespace.--as-group stringsGroup to impersonate for the operation, this flag can be repeated to specify multiple groups.--as-uid stringUID to impersonate for the operation.--cache-dir string Default: "$HOME/.kube/cache"Default cache directory--certificate-authority stringPath to a cert file for the certificate authority--client-certificate stringPath to a client certificate file for TLS--client-key stringPath to a client key file for TLS--cluster stringThe name of the kubeconfig cluster to use--context stringThe name of the kubeconfig context to use--disable-compressionIf true, opt-out of response compression for all requests to the server--insecure-skip-tls-verifyIf true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure--kubeconfig stringPath to the kubeconfig file to use for CLI requests.--kuberc stringPath to the kuberc file to use for preferences. This can be disabled by exporting KUBECTL_KUBERC=false feature gate or turning off the feature KUBERC=off.--match-server-versionRequire server version to match client version-n, --namespace stringIf present, the namespace scope for this CLI request--password stringPassword for basic authentication to the API server--profile string Default: "none"Name of profile to capture. One of (none|cpu|heap|goroutine|threadcreate|block|mutex)--profile-output string Default: "profile.pprof"Name of the file to write the profile to--request-timeout string Default: "0"The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests.-s, --server stringThe address and port of the Kubernetes API server--storage-driver-buffer-duration duration Default: 1m0sWrites in the storage driver will be buffered for this duration, and committed to the non memory backends as a single transaction--storage-driver-db string Default: "cadvisor"database name--storage-driver-host string Default: "localhost:8086"database host:port--storage-driver-password string Default: "root"database password--storage-driver-secureuse secure connection with database--storage-driver-table string Default: "stats"table name--storage-driver-user string Default: "root"database username--tls-server-name stringServer name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used--token stringBearer token for authentication to the API server--user stringThe name of the kubeconfig user to use--username stringUsername for basic authentication to the API server--version version[=true]--version, --version=raw prints version information and quits; --version=vX.Y.Z... sets the reported version--warnings-as-errorsTreat warnings received from the server as errors and exit with a non-zero exit codeSee Alsokubectl certificate - Modify certificate resourcesThis page is automatically generated.If you plan to report an issue with this page, mention that the page is auto-generated in your issue description. The fix may need to happen elsewhere in the Kubernetes project.


# Service Resources

Service ResourcesServiceService is a named abstraction of software service (for example, mysql) consisting of local port (for example 3306) that the proxy listens on, and the selector that determines which pods will answer requests sent through the proxy.EndpointsEndpoints is a collection of endpoints that implement the actual service.EndpointSliceEndpointSlice represents a set of service endpoints.IngressIngress is a collection of rules that allow inbound connections to reach the endpoints defined by a backend.IngressClassIngressClass represents the class of the Ingress, referenced by the Ingress Spec.This page is automatically generated.If you plan to report an issue with this page, mention that the page is auto-generated in your issue description. The fix may need to happen elsewhere in the Kubernetes project.