# Running ZooKeeper, A Distributed System Coordinator

Running ZooKeeper, A Distributed System CoordinatorThis tutorial demonstrates running Apache Zookeeper on Kubernetes using StatefulSets, PodDisruptionBudgets, and PodAntiAffinity.Before you beginBefore starting this tutorial, you should be familiar with the following Kubernetes concepts:PodsCluster DNSHeadless ServicesPersistentVolumesPersistentVolume ProvisioningStatefulSetsPodDisruptionBudgetsPodAntiAffinitykubectl CLIYou must have a cluster with at least four nodes, and each node requires at least 2 CPUs and 4 GiB of memory. In this tutorial you will cordon and drain the cluster's nodes. This means that the cluster will terminate and evict all Pods on its nodes, and the nodes will temporarily become unschedulable. You should use a dedicated cluster for this tutorial, or you should ensure that the disruption you cause will not interfere with other tenants.This tutorial assumes that you have configured your cluster to dynamically provision PersistentVolumes. If your cluster is not configured to do so, you will have to manually provision three 20 GiB volumes before starting this tutorial.ObjectivesAfter this tutorial, you will know the following.How to deploy a ZooKeeper ensemble using StatefulSet.How to consistently configure the ensemble.How to spread the deployment of ZooKeeper servers in the ensemble.How to use PodDisruptionBudgets to ensure service availability during planned maintenance.ZooKeeperApache ZooKeeper is a distributed, open-source coordination service for distributed applications. ZooKeeper allows you to read, write, and observe updates to data. Data are organized in a file system like hierarchy and replicated to all ZooKeeper servers in the ensemble (a set of ZooKeeper servers). All operations on data are atomic and sequentially consistent. ZooKeeper ensures this by using the Zab consensus protocol to replicate a state machine across all servers in the ensemble.The ensemble uses the Zab protocol to elect a leader, and the ensemble cannot write data until that election is complete. Once complete, the ensemble uses Zab to ensure that it replicates all writes to a quorum before it acknowledges and makes them visible to clients. Without respect to weighted quorums, a quorum is a majority component of the ensemble containing the current leader. For instance, if the ensemble has three servers, a component that contains the leader and one other server constitutes a quorum. If the ensemble can not achieve a quorum, the ensemble cannot write data.ZooKeeper servers keep their entire state machine in memory, and write every mutation to a durable WAL (Write Ahead Log) on storage media. When a server crashes, it can recover its previous state by replaying the WAL. To prevent the WAL from growing without bound, ZooKeeper servers will periodically snapshot them in memory state to storage media. These snapshots can be loaded directly into memory, and all WAL entries that preceded the snapshot may be discarded.Creating a ZooKeeper ensembleThe manifest below contains a Headless Service, a Service, a PodDisruptionBudget, and a StatefulSet.application/zookeeper/zookeeper.yaml apiVersion: v1 kind: Service metadata: name: zk-hs labels: app: zk spec: ports: - port: 2888 name: server - port: 3888 name: leader-election clusterIP: None selector: app: zk --- apiVersion: v1 kind: Service metadata: name: zk-cs labels: app: zk spec: ports: - port: 2181 name: client selector: app: zk --- apiVersion: policy/v1 kind: PodDisruptionBudget metadata: name: zk-pdb spec: selector: matchLabels: app: zk maxUnavailable: 1 --- apiVersion: apps/v1 kind: StatefulSet metadata: name: zk spec: selector: matchLabels: app: zk serviceName: zk-hs replicas: 3 updateStrategy: type: RollingUpdate podManagementPolicy: OrderedReady template: metadata: labels: app: zk spec: affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchExpressions: - key: "app" operator: In values: - zk topologyKey: "kubernetes.io/hostname" containers: - name: kubernetes-zookeeper imagePullPolicy: Always image: "registry.k8s.io/kubernetes-zookeeper:1.0-3.4.10" resources: requests: memory: "1Gi" cpu: "0.5" ports: - containerPort: 2181 name: client - containerPort: 2888 name: server - containerPort: 3888 name: leader-election command: - sh - -c - "start-zookeeper \ --servers=3 \ --data_dir=/var/lib/zookeeper/data \ --data_log_dir=/var/lib/zookeeper/data/log \ --conf_dir=/opt/zookeeper/conf \ --client_port=2181 \ --election_port=3888 \ --server_port=2888 \ --tick_time=2000 \ --init_limit=10 \ --sync_limit=5 \ --heap=512M \ --max_client_cnxns=60 \ --snap_retain_count=3 \ --purge_interval=12 \ --max_session_timeout=40000 \ --min_session_timeout=4000 \ --log_level=INFO" readinessProbe: exec: command: - sh - -c - "zookeeper-ready 2181" initialDelaySeconds: 10 timeoutSeconds: 5 livenessProbe: exec: command: - sh - -c - "zookeeper-ready 2181" initialDelaySeconds: 10 timeoutSeconds: 5 volumeMounts: - name: datadir mountPath: /var/lib/zookeeper securityContext: runAsUser: 1000 fsGroup: 1000 volumeClaimTemplates: - metadata: name: datadir spec: accessModes: [ "ReadWriteOnce" ] resources: requests: storage: 10Gi Open a terminal, and use the kubectl apply command to create the manifest.kubectl apply -f https://k8s.io/examples/application/zookeeper/zookeeper.yaml This creates the zk-hs Headless Service, the zk-cs Service, the zk-pdb PodDisruptionBudget, and the zk StatefulSet.service/zk-hs created service/zk-cs created poddisruptionbudget.policy/zk-pdb created statefulset.apps/zk created Use kubectl get to watch the StatefulSet controller create the StatefulSet's Pods.kubectl get pods -w -l app=zk Once the zk-2 Pod is Running and Ready, use CTRL-C to terminate kubectl.NAME READY STATUS RESTARTS AGE zk-0 0/1 Pending 0 0s zk-0 0/1 Pending 0 0s zk-0 0/1 ContainerCreating 0 0s zk-0 0/1 Running 0 19s zk-0 1/1 Running 0 40s zk-1 0/1 Pending 0 0s zk-1 0/1 Pending 0 0s zk-1 0/1 ContainerCreating 0 0s zk-1 0/1 Running 0 18s zk-1 1/1 Running 0 40s zk-2 0/1 Pending 0 0s zk-2 0/1 Pending 0 0s zk-2 0/1 ContainerCreating 0 0s zk-2 0/1 Running 0 19s zk-2 1/1 Running 0 40s The StatefulSet controller creates three Pods, and each Pod has a container with a ZooKeeper server.Facilitating leader electionBecause there is no terminating algorithm for electing a leader in an anonymous network, Zab requires explicit membership configuration to perform leader election. Each server in the ensemble needs to have a unique identifier, all servers need to know the global set of identifiers, and each identifier needs to be associated with a network address.Use kubectl exec to get the hostnames of the Pods in the zk StatefulSet.for i in 0 1 2; do kubectl exec zk-$i -- hostname; done The StatefulSet controller provides each Pod with a unique hostname based on its ordinal index. The hostnames take the form of <statefulset name>-<ordinal index>. Because the replicas field of the zk StatefulSet is set to 3, the Set's controller creates three Pods with their hostnames set to zk-0, zk-1, and zk-2.zk-0 zk-1 zk-2 The servers in a ZooKeeper ensemble use natural numbers as unique identifiers, and store each server's identifier in a file called myid in the server's data directory.To examine the contents of the myid file for each server use the following command.for i in 0 1 2; do echo "myid zk-$i";kubectl exec zk-$i -- cat /var/lib/zookeeper/data/myid; done Because the identifiers are natural numbers and the ordinal indices are non-negative integers, you can generate an identifier by adding 1 to the ordinal.myid zk-0 1 myid zk-1 2 myid zk-2 3 To get the Fully Qualified Domain Name (FQDN) of each Pod in the zk StatefulSet use the following command.for i in 0 1 2; do kubectl exec zk-$i -- hostname -f; done The zk-hs Service creates a domain for all of the Pods, zk-hs.default.svc.cluster.local.zk-0.zk-hs.default.svc.cluster.local zk-1.zk-hs.default.svc.cluster.local zk-2.zk-hs.default.svc.cluster.local The A records in Kubernetes DNS resolve the FQDNs to the Pods' IP addresses. If Kubernetes reschedules the Pods, it will update the A records with the Pods' new IP addresses, but the A records names will not change.ZooKeeper stores its application configuration in a file named zoo.cfg. Use kubectl exec to view the contents of the zoo.cfg file in the zk-0 Pod.kubectl exec zk-0 -- cat /opt/zookeeper/conf/zoo.cfg In the server.1, server.2, and server.3 properties at the bottom of the file, the 1, 2, and 3 correspond to the identifiers in the ZooKeeper servers' myid files. They are set to the FQDNs for the Pods in the zk StatefulSet.clientPort=2181 dataDir=/var/lib/zookeeper/data dataLogDir=/var/lib/zookeeper/log tickTime=2000 initLimit=10 syncLimit=2000 maxClientCnxns=60 minSessionTimeout= 4000 maxSessionTimeout= 40000 autopurge.snapRetainCount=3 autopurge.purgeInterval=0 server.1=zk-0.zk-hs.default.svc.cluster.local:2888:3888 server.2=zk-1.zk-hs.default.svc.cluster.local:2888:3888 server.3=zk-2.zk-hs.default.svc.cluster.local:2888:3888 Achieving consensusConsensus protocols require that the identifiers of each participant be unique. No two participants in the Zab protocol should claim the same unique identifier. This is necessary to allow the processes in the system to agree on which processes have committed which data. If two Pods are launched with the same ordinal, two ZooKeeper servers would both identify themselves as the same server.kubectl get pods -w -l app=zk NAME READY STATUS RESTARTS AGE zk-0 0/1 Pending 0 0s zk-0 0/1 Pending 0 0s zk-0 0/1 ContainerCreating 0 0s zk-0 0/1 Running 0 19s zk-0 1/1 Running 0 40s zk-1 0/1 Pending 0 0s zk-1 0/1 Pending 0 0s zk-1 0/1 ContainerCreating 0 0s zk-1 0/1 Running 0 18s zk-1 1/1 Running 0 40s zk-2 0/1 Pending 0 0s zk-2 0/1 Pending 0 0s zk-2 0/1 ContainerCreating 0 0s zk-2 0/1 Running 0 19s zk-2 1/1 Running 0 40s The A records for each Pod are entered when the Pod becomes Ready. Therefore, the FQDNs of the ZooKeeper servers will resolve to a single endpoint, and that endpoint will be the unique ZooKeeper server claiming the identity configured in its myid file.zk-0.zk-hs.default.svc.cluster.local zk-1.zk-hs.default.svc.cluster.local zk-2.zk-hs.default.svc.cluster.local This ensures that the servers properties in the ZooKeepers' zoo.cfg files represents a correctly configured ensemble.server.1=zk-0.zk-hs.default.svc.cluster.local:2888:3888 server.2=zk-1.zk-hs.default.svc.cluster.local:2888:3888 server.3=zk-2.zk-hs.default.svc.cluster.local:2888:3888 When the servers use the Zab protocol to attempt to commit a value, they will either achieve consensus and commit the value (if leader election has succeeded and at least two of the Pods are Running and Ready), or they will fail to do so (if either of the conditions are not met). No state will arise where one server acknowledges a write on behalf of another.Sanity testing the ensembleThe most basic sanity test is to write data to one ZooKeeper server and to read the data from another.The command below executes the zkCli.sh script to write world to the path /hello on the zk-0 Pod in the ensemble.kubectl exec zk-0 -- zkCli.sh create /hello world WATCHER:: WatchedEvent state:SyncConnected type:None path:null Created /hello To get the data from the zk-1 Pod use the following command.kubectl exec zk-1 -- zkCli.sh get /hello The data that you created on zk-0 is available on all the servers in the ensemble.WATCHER:: WatchedEvent state:SyncConnected type:None path:null world cZxid = 0x100000002 ctime = Thu Dec 08 15:13:30 UTC 2016 mZxid = 0x100000002 mtime = Thu Dec 08 15:13:30 UTC 2016 pZxid = 0x100000002 cversion = 0 dataVersion = 0 aclVersion = 0 ephemeralOwner = 0x0 dataLength = 5 numChildren = 0 Providing durable storageAs mentioned in the ZooKeeper Basics section, ZooKeeper commits all entries to a durable WAL, and periodically writes snapshots in memory state, to storage media. Using WALs to provide durability is a common technique for applications that use consensus protocols to achieve a replicated state machine.Use the kubectl delete command to delete the zk StatefulSet.kubectl delete statefulset zk statefulset.apps "zk" deleted Watch the termination of the Pods in the StatefulSet.kubectl get pods -w -l app=zk When zk-0 if fully terminated, use CTRL-C to terminate kubectl.zk-2 1/1 Terminating 0 9m zk-0 1/1 Terminating 0 11m zk-1 1/1 Terminating 0 10m zk-2 0/1 Terminating 0 9m zk-2 0/1 Terminating 0 9m zk-2 0/1 Terminating 0 9m zk-1 0/1 Terminating 0 10m zk-1 0/1 Terminating 0 10m zk-1 0/1 Terminating 0 10m zk-0 0/1 Terminating 0 11m zk-0 0/1 Terminating 0 11m zk-0 0/1 Terminating 0 11m Reapply the manifest in zookeeper.yaml.kubectl apply -f https://k8s.io/examples/application/zookeeper/zookeeper.yaml This creates the zk StatefulSet object, but the other API objects in the manifest are not modified because they already exist.Watch the StatefulSet controller recreate the StatefulSet's Pods.kubectl get pods -w -l app=zk Once the zk-2 Pod is Running and Ready, use CTRL-C to terminate kubectl.NAME READY STATUS RESTARTS AGE zk-0 0/1 Pending 0 0s zk-0 0/1 Pending 0 0s zk-0 0/1 ContainerCreating 0 0s zk-0 0/1 Running 0 19s zk-0 1/1 Running 0 40s zk-1 0/1 Pending 0 0s zk-1 0/1 Pending 0 0s zk-1 0/1 ContainerCreating 0 0s zk-1 0/1 Running 0 18s zk-1 1/1 Running 0 40s zk-2 0/1 Pending 0 0s zk-2 0/1 Pending 0 0s zk-2 0/1 ContainerCreating 0 0s zk-2 0/1 Running 0 19s zk-2 1/1 Running 0 40s Use the command below to get the value you entered during the sanity test, from the zk-2 Pod.kubectl exec zk-2 zkCli.sh get /hello Even though you terminated and recreated all of the Pods in the zk StatefulSet, the ensemble still serves the original value.WATCHER:: WatchedEvent state:SyncConnected type:None path:null world cZxid = 0x100000002 ctime = Thu Dec 08 15:13:30 UTC 2016 mZxid = 0x100000002 mtime = Thu Dec 08 15:13:30 UTC 2016 pZxid = 0x100000002 cversion = 0 dataVersion = 0 aclVersion = 0 ephemeralOwner = 0x0 dataLength = 5 numChildren = 0 The volumeClaimTemplates field of the zk StatefulSet's spec specifies a PersistentVolume provisioned for each Pod.volumeClaimTemplates: - metadata: name: datadir annotations: volume.alpha.kubernetes.io/storage-class: anything spec: accessModes: [ "ReadWriteOnce" ] resources: requests: storage: 20Gi The StatefulSet controller generates a PersistentVolumeClaim for each Pod in the StatefulSet.Use the following command to get the StatefulSet's PersistentVolumeClaims.kubectl get pvc -l app=zk When the StatefulSet recreated its Pods, it remounts the Pods' PersistentVolumes.NAME STATUS VOLUME CAPACITY ACCESSMODES AGE datadir-zk-0 Bound pvc-bed742cd-bcb1-11e6-994f-42010a800002 20Gi RWO 1h datadir-zk-1 Bound pvc-bedd27d2-bcb1-11e6-994f-42010a800002 20Gi RWO 1h datadir-zk-2 Bound pvc-bee0817e-bcb1-11e6-994f-42010a800002 20Gi RWO 1h The volumeMounts section of the StatefulSet's container template mounts the PersistentVolumes in the ZooKeeper servers' data directories.volumeMounts: - name: datadir mountPath: /var/lib/zookeeper When a Pod in the zk StatefulSet is (re)scheduled, it will always have the same PersistentVolume mounted to the ZooKeeper server's data directory. Even when the Pods are rescheduled, all the writes made to the ZooKeeper servers' WALs, and all their snapshots, remain durable.Ensuring consistent configurationAs noted in the Facilitating Leader Election and Achieving Consensus sections, the servers in a ZooKeeper ensemble require consistent configuration to elect a leader and form a quorum. They also require consistent configuration of the Zab protocol in order for the protocol to work correctly over a network. In our example we achieve consistent configuration by embedding the configuration directly into the manifest.Get the zk StatefulSet.kubectl get sts zk -o yaml … command: - sh - -c - "start-zookeeper \ --servers=3 \ --data_dir=/var/lib/zookeeper/data \ --data_log_dir=/var/lib/zookeeper/data/log \ --conf_dir=/opt/zookeeper/conf \ --client_port=2181 \ --election_port=3888 \ --server_port=2888 \ --tick_time=2000 \ --init_limit=10 \ --sync_limit=5 \ --heap=512M \ --max_client_cnxns=60 \ --snap_retain_count=3 \ --purge_interval=12 \ --max_session_timeout=40000 \ --min_session_timeout=4000 \ --log_level=INFO" … The command used to start the ZooKeeper servers passed the configuration as command line parameter. You can also use environment variables to pass configuration to the ensemble.Configuring loggingOne of the files generated by the zkGenConfig.sh script controls ZooKeeper's logging. ZooKeeper uses Log4j, and, by default, it uses a time and size based rolling file appender for its logging configuration.Use the command below to get the logging configuration from one of Pods in the zk StatefulSet.kubectl exec zk-0 cat /usr/etc/zookeeper/log4j.properties The logging configuration below will cause the ZooKeeper process to write all of its logs to the standard output file stream.zookeeper.root.logger=CONSOLE zookeeper.console.threshold=INFO log4j.rootLogger=${zookeeper.root.logger} log4j.appender.CONSOLE=org.apache.log4j.ConsoleAppender log4j.appender.CONSOLE.Threshold=${zookeeper.console.threshold} log4j.appender.CONSOLE.layout=org.apache.log4j.PatternLayout log4j.appender.CONSOLE.layout.ConversionPattern=%d{ISO8601} [myid:%X{myid}] - %-5p [%t:%C{1}@%L] - %m%n This is the simplest possible way to safely log inside the container. Because the applications write logs to standard out, Kubernetes will handle log rotation for you. Kubernetes also implements a sane retention policy that ensures application logs written to standard out and standard error do not exhaust local storage media.Use kubectl logs to retrieve the last 20 log lines from one of the Pods.kubectl logs zk-0 --tail 20 You can view application logs written to standard out or standard error using kubectl logs and from the Kubernetes Dashboard.2016-12-06 19:34:16,236 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@827] - Processing ruok command from /127.0.0.1:52740 2016-12-06 19:34:16,237 [myid:1] - INFO [Thread-1136:NIOServerCnxn@1008] - Closed socket connection for client /127.0.0.1:52740 (no session established for client) 2016-12-06 19:34:26,155 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@192] - Accepted socket connection from /127.0.0.1:52749 2016-12-06 19:34:26,155 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@827] - Processing ruok command from /127.0.0.1:52749 2016-12-06 19:34:26,156 [myid:1] - INFO [Thread-1137:NIOServerCnxn@1008] - Closed socket connection for client /127.0.0.1:52749 (no session established for client) 2016-12-06 19:34:26,222 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@192] - Accepted socket connection from /127.0.0.1:52750 2016-12-06 19:34:26,222 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@827] - Processing ruok command from /127.0.0.1:52750 2016-12-06 19:34:26,226 [myid:1] - INFO [Thread-1138:NIOServerCnxn@1008] - Closed socket connection for client /127.0.0.1:52750 (no session established for client) 2016-12-06 19:34:36,151 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@192] - Accepted socket connection from /127.0.0.1:52760 2016-12-06 19:34:36,152 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@827] - Processing ruok command from /127.0.0.1:52760 2016-12-06 19:34:36,152 [myid:1] - INFO [Thread-1139:NIOServerCnxn@1008] - Closed socket connection for client /127.0.0.1:52760 (no session established for client) 2016-12-06 19:34:36,230 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@192] - Accepted socket connection from /127.0.0.1:52761 2016-12-06 19:34:36,231 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@827] - Processing ruok command from /127.0.0.1:52761 2016-12-06 19:34:36,231 [myid:1] - INFO [Thread-1140:NIOServerCnxn@1008] - Closed socket connection for client /127.0.0.1:52761 (no session established for client) 2016-12-06 19:34:46,149 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@192] - Accepted socket connection from /127.0.0.1:52767 2016-12-06 19:34:46,149 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@827] - Processing ruok command from /127.0.0.1:52767 2016-12-06 19:34:46,149 [myid:1] - INFO [Thread-1141:NIOServerCnxn@1008] - Closed socket connection for client /127.0.0.1:52767 (no session established for client) 2016-12-06 19:34:46,230 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@192] - Accepted socket connection from /127.0.0.1:52768 2016-12-06 19:34:46,230 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@827] - Processing ruok command from /127.0.0.1:52768 2016-12-06 19:34:46,230 [myid:1] - INFO [Thread-1142:NIOServerCnxn@1008] - Closed socket connection for client /127.0.0.1:52768 (no session established for client) Kubernetes integrates with many logging solutions. You can choose a logging solution that best fits your cluster and applications. For cluster-level logging and aggregation, consider deploying a sidecar container to rotate and ship your logs.Configuring a non-privileged userThe best practices to allow an application to run as a privileged user inside of a container are a matter of debate. If your organization requires that applications run as a non-privileged user you can use a SecurityContext to control the user that the entry point runs as.The zk StatefulSet's Pod template contains a SecurityContext.securityContext: runAsUser: 1000 fsGroup: 1000 In the Pods' containers, UID 1000 corresponds to the zookeeper user and GID 1000 corresponds to the zookeeper group.Get the ZooKeeper process information from the zk-0 Pod.kubectl exec zk-0 -- ps -elf As the runAsUser field of the securityContext object is set to 1000, instead of running as root, the ZooKeeper process runs as the zookeeper user.F S UID PID PPID C PRI NI ADDR SZ WCHAN STIME TTY TIME CMD 4 S zookeep+ 1 0 0 80 0 - 1127 - 20:46 ? 00:00:00 sh -c zkGenConfig.sh && zkServer.sh start-foreground 0 S zookeep+ 27 1 0 80 0 - 1155556 - 20:46 ? 00:00:19 /usr/lib/jvm/java-8-openjdk-amd64/bin/java -Dzookeeper.log.dir=/var/log/zookeeper -Dzookeeper.root.logger=INFO,CONSOLE -cp /usr/bin/../build/classes:/usr/bin/../build/lib/*.jar:/usr/bin/../share/zookeeper/zookeeper-3.4.9.jar:/usr/bin/../share/zookeeper/slf4j-log4j12-1.6.1.jar:/usr/bin/../share/zookeeper/slf4j-api-1.6.1.jar:/usr/bin/../share/zookeeper/netty-3.10.5.Final.jar:/usr/bin/../share/zookeeper/log4j-1.2.16.jar:/usr/bin/../share/zookeeper/jline-0.9.94.jar:/usr/bin/../src/java/lib/*.jar:/usr/bin/../etc/zookeeper: -Xmx2G -Xms2G -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.local.only=false org.apache.zookeeper.server.quorum.QuorumPeerMain /usr/bin/../etc/zookeeper/zoo.cfg By default, when the Pod's PersistentVolumes is mounted to the ZooKeeper server's data directory, it is only accessible by the root user. This configuration prevents the ZooKeeper process from writing to its WAL and storing its snapshots.Use the command below to get the file permissions of the ZooKeeper data directory on the zk-0 Pod.kubectl exec -ti zk-0 -- ls -ld /var/lib/zookeeper/data Because the fsGroup field of the securityContext object is set to 1000, the ownership of the Pods' PersistentVolumes is set to the zookeeper group, and the ZooKeeper process is able to read and write its data.drwxr-sr-x 3 zookeeper zookeeper 4096 Dec 5 20:45 /var/lib/zookeeper/data Managing the ZooKeeper processThe ZooKeeper documentation mentions that "You will want to have a supervisory process that manages each of your ZooKeeper server processes (JVM)." Utilizing a watchdog (supervisory process) to restart failed processes in a distributed system is a common pattern. When deploying an application in Kubernetes, rather than using an external utility as a supervisory process, you should use Kubernetes as the watchdog for your application.Updating the ensembleThe zk StatefulSet is configured to use the RollingUpdate update strategy.You can use kubectl patch to update the number of cpus allocated to the servers.kubectl patch sts zk --type='json' -p='[{"op": "replace", "path": "/spec/template/spec/containers/0/resources/requests/cpu", "value":"0.3"}]' statefulset.apps/zk patched Use kubectl rollout status to watch the status of the update.kubectl rollout status sts/zk waiting for statefulset rolling update to complete 0 pods at revision zk-5db4499664... Waiting for 1 pods to be ready... Waiting for 1 pods to be ready... waiting for statefulset rolling update to complete 1 pods at revision zk-5db4499664... Waiting for 1 pods to be ready... Waiting for 1 pods to be ready... waiting for statefulset rolling update to complete 2 pods at revision zk-5db4499664... Waiting for 1 pods to be ready... Waiting for 1 pods to be ready... statefulset rolling update complete 3 pods at revision zk-5db4499664... This terminates the Pods, one at a time, in reverse ordinal order, and recreates them with the new configuration. This ensures that quorum is maintained during a rolling update.Use the kubectl rollout history command to view a history or previous configurations.kubectl rollout history sts/zk The output is similar to this:statefulsets "zk" REVISION 1 2 Use the kubectl rollout undo command to roll back the modification.kubectl rollout undo sts/zk The output is similar to this:statefulset.apps/zk rolled back Handling process failureRestart Policies control how Kubernetes handles process failures for the entry point of the container in a Pod. For Pods in a StatefulSet, the only appropriate RestartPolicy is Always, and this is the default value. For stateful applications you should never override the default policy.Use the following command to examine the process tree for the ZooKeeper server running in the zk-0 Pod.kubectl exec zk-0 -- ps -ef The command used as the container's entry point has PID 1, and the ZooKeeper process, a child of the entry point, has PID 27.UID PID PPID C STIME TTY TIME CMD zookeep+ 1 0 0 15:03 ? 00:00:00 sh -c zkGenConfig.sh && zkServer.sh start-foreground zookeep+ 27 1 0 15:03 ? 00:00:03 /usr/lib/jvm/java-8-openjdk-amd64/bin/java -Dzookeeper.log.dir=/var/log/zookeeper -Dzookeeper.root.logger=INFO,CONSOLE -cp /usr/bin/../build/classes:/usr/bin/../build/lib/*.jar:/usr/bin/../share/zookeeper/zookeeper-3.4.9.jar:/usr/bin/../share/zookeeper/slf4j-log4j12-1.6.1.jar:/usr/bin/../share/zookeeper/slf4j-api-1.6.1.jar:/usr/bin/../share/zookeeper/netty-3.10.5.Final.jar:/usr/bin/../share/zookeeper/log4j-1.2.16.jar:/usr/bin/../share/zookeeper/jline-0.9.94.jar:/usr/bin/../src/java/lib/*.jar:/usr/bin/../etc/zookeeper: -Xmx2G -Xms2G -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.local.only=false org.apache.zookeeper.server.quorum.QuorumPeerMain /usr/bin/../etc/zookeeper/zoo.cfg In another terminal watch the Pods in the zk StatefulSet with the following command.kubectl get pod -w -l app=zk In another terminal, terminate the ZooKeeper process in Pod zk-0 with the following command.kubectl exec zk-0 -- pkill java The termination of the ZooKeeper process caused its parent process to terminate. Because the RestartPolicy of the container is Always, it restarted the parent process.NAME READY STATUS RESTARTS AGE zk-0 1/1 Running 0 21m zk-1 1/1 Running 0 20m zk-2 1/1 Running 0 19m NAME READY STATUS RESTARTS AGE zk-0 0/1 Error 0 29m zk-0 0/1 Running 1 29m zk-0 1/1 Running 1 29m If your application uses a script (such as zkServer.sh) to launch the process that implements the application's business logic, the script must terminate with the child process. This ensures that Kubernetes will restart the application's container when the process implementing the application's business logic fails.Testing for livenessConfiguring your application to restart failed processes is not enough to keep a distributed system healthy. There are scenarios where a system's processes can be both alive and unresponsive, or otherwise unhealthy. You should use liveness probes to notify Kubernetes that your application's processes are unhealthy and it should restart them.The Pod template for the zk StatefulSet specifies a liveness probe. livenessProbe: exec: command: - sh - -c - "zookeeper-ready 2181" initialDelaySeconds: 15 timeoutSeconds: 5 The probe calls a bash script that uses the ZooKeeper ruok four letter word to test the server's health.OK=$(echo ruok | nc 127.0.0.1 $1) if [ "$OK" == "imok" ]; then exit 0 else exit 1 fi In one terminal window, use the following command to watch the Pods in the zk StatefulSet.kubectl get pod -w -l app=zk In another window, using the following command to delete the zookeeper-ready script from the file system of Pod zk-0.kubectl exec zk-0 -- rm /opt/zookeeper/bin/zookeeper-ready When the liveness probe for the ZooKeeper process fails, Kubernetes will automatically restart the process for you, ensuring that unhealthy processes in the ensemble are restarted.kubectl get pod -w -l app=zk NAME READY STATUS RESTARTS AGE zk-0 1/1 Running 0 1h zk-1 1/1 Running 0 1h zk-2 1/1 Running 0 1h NAME READY STATUS RESTARTS AGE zk-0 0/1 Running 0 1h zk-0 0/1 Running 1 1h zk-0 1/1 Running 1 1h Testing for readinessReadiness is not the same as liveness. If a process is alive, it is scheduled and healthy. If a process is ready, it is able to process input. Liveness is a necessary, but not sufficient, condition for readiness. There are cases, particularly during initialization and termination, when a process can be alive but not ready.If you specify a readiness probe, Kubernetes will ensure that your application's processes will not receive network traffic until their readiness checks pass.For a ZooKeeper server, liveness implies readiness. Therefore, the readiness probe from the zookeeper.yaml manifest is identical to the liveness probe. readinessProbe: exec: command: - sh - -c - "zookeeper-ready 2181" initialDelaySeconds: 15 timeoutSeconds: 5 Even though the liveness and readiness probes are identical, it is important to specify both. This ensures that only healthy servers in the ZooKeeper ensemble receive network traffic.Tolerating Node failureZooKeeper needs a quorum of servers to successfully commit mutations to data. For a three server ensemble, two servers must be healthy for writes to succeed. In quorum based systems, members are deployed across failure domains to ensure availability. To avoid an outage, due to the loss of an individual machine, best practices preclude co-locating multiple instances of the application on the same machine.By default, Kubernetes may co-locate Pods in a StatefulSet on the same node. For the three server ensemble you created, if two servers are on the same node, and that node fails, the clients of your ZooKeeper service will experience an outage until at least one of the Pods can be rescheduled.You should always provision additional capacity to allow the processes of critical systems to be rescheduled in the event of node failures. If you do so, then the outage will only last until the Kubernetes scheduler reschedules one of the ZooKeeper servers. However, if you want your service to tolerate node failures with no downtime, you should set podAntiAffinity.Use the command below to get the nodes for Pods in the zk StatefulSet.for i in 0 1 2; do kubectl get pod zk-$i --template {{.spec.nodeName}}; echo ""; done All of the Pods in the zk StatefulSet are deployed on different nodes.kubernetes-node-cxpk kubernetes-node-a5aq kubernetes-node-2g2d This is because the Pods in the zk StatefulSet have a PodAntiAffinity specified.affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchExpressions: - key: "app" operator: In values: - zk topologyKey: "kubernetes.io/hostname" The requiredDuringSchedulingIgnoredDuringExecution field tells the Kubernetes Scheduler that it should never co-locate two Pods which have app label as zk in the domain defined by the topologyKey. The topologyKey kubernetes.io/hostname indicates that the domain is an individual node. Using different rules, labels, and selectors, you can extend this technique to spread your ensemble across physical, network, and power failure domains.Surviving maintenanceIn this section you will cordon and drain nodes. If you are using this tutorial on a shared cluster, be sure that this will not adversely affect other tenants.The previous section showed you how to spread your Pods across nodes to survive unplanned node failures, but you also need to plan for temporary node failures that occur due to planned maintenance.Use this command to get the nodes in your cluster.kubectl get nodes This tutorial assumes a cluster with at least four nodes. If the cluster has more than four, use kubectl cordon to cordon all but four nodes. Constraining to four nodes will ensure Kubernetes encounters affinity and PodDisruptionBudget constraints when scheduling zookeeper Pods in the following maintenance simulation.kubectl cordon <node-name> Use this command to get the zk-pdb PodDisruptionBudget.kubectl get pdb zk-pdb The max-unavailable field indicates to Kubernetes that at most one Pod from zk StatefulSet can be unavailable at any time.NAME MIN-AVAILABLE MAX-UNAVAILABLE ALLOWED-DISRUPTIONS AGE zk-pdb N/A 1 1 In one terminal, use this command to watch the Pods in the zk StatefulSet.kubectl get pods -w -l app=zk In another terminal, use this command to get the nodes that the Pods are currently scheduled on.for i in 0 1 2; do kubectl get pod zk-$i --template {{.spec.nodeName}}; echo ""; done The output is similar to this:kubernetes-node-pb41 kubernetes-node-ixsl kubernetes-node-i4c4 Use kubectl drain to cordon and drain the node on which the zk-0 Pod is scheduled.kubectl drain $(kubectl get pod zk-0 --template {{.spec.nodeName}}) --ignore-daemonsets --force --delete-emptydir-data The output is similar to this:node "kubernetes-node-pb41" cordoned WARNING: Deleting pods not managed by ReplicationController, ReplicaSet, Job, or DaemonSet: fluentd-cloud-logging-kubernetes-node-pb41, kube-proxy-kubernetes-node-pb41; Ignoring DaemonSet-managed pods: node-problem-detector-v0.1-o5elz pod "zk-0" deleted node "kubernetes-node-pb41" drained As there are four nodes in your cluster, kubectl drain, succeeds and the zk-0 is rescheduled to another node.NAME READY STATUS RESTARTS AGE zk-0 1/1 Running 2 1h zk-1 1/1 Running 0 1h zk-2 1/1 Running 0 1h NAME READY STATUS RESTARTS AGE zk-0 1/1 Terminating 2 2h zk-0 0/1 Terminating 2 2h zk-0 0/1 Terminating 2 2h zk-0 0/1 Terminating 2 2h zk-0 0/1 Pending 0 0s zk-0 0/1 Pending 0 0s zk-0 0/1 ContainerCreating 0 0s zk-0 0/1 Running 0 51s zk-0 1/1 Running 0 1m Keep watching the StatefulSet's Pods in the first terminal and drain the node on which zk-1 is scheduled.kubectl drain $(kubectl get pod zk-1 --template {{.spec.nodeName}}) --ignore-daemonsets --force --delete-emptydir-data The output is similar to this:"kubernetes-node-ixsl" cordoned WARNING: Deleting pods not managed by ReplicationController, ReplicaSet, Job, or DaemonSet: fluentd-cloud-logging-kubernetes-node-ixsl, kube-proxy-kubernetes-node-ixsl; Ignoring DaemonSet-managed pods: node-problem-detector-v0.1-voc74 pod "zk-1" deleted node "kubernetes-node-ixsl" drained The zk-1 Pod cannot be scheduled because the zk StatefulSet contains a PodAntiAffinity rule preventing co-location of the Pods, and as only two nodes are schedulable, the Pod will remain in a Pending state.kubectl get pods -w -l app=zk The output is similar to this:NAME READY STATUS RESTARTS AGE zk-0 1/1 Running 2 1h zk-1 1/1 Running 0 1h zk-2 1/1 Running 0 1h NAME READY STATUS RESTARTS AGE zk-0 1/1 Terminating 2 2h zk-0 0/1 Terminating 2 2h zk-0 0/1 Terminating 2 2h zk-0 0/1 Terminating 2 2h zk-0 0/1 Pending 0 0s zk-0 0/1 Pending 0 0s zk-0 0/1 ContainerCreating 0 0s zk-0 0/1 Running 0 51s zk-0 1/1 Running 0 1m zk-1 1/1 Terminating 0 2h zk-1 0/1 Terminating 0 2h zk-1 0/1 Terminating 0 2h zk-1 0/1 Terminating 0 2h zk-1 0/1 Pending 0 0s zk-1 0/1 Pending 0 0s Continue to watch the Pods of the StatefulSet, and drain the node on which zk-2 is scheduled.kubectl drain $(kubectl get pod zk-2 --template {{.spec.nodeName}}) --ignore-daemonsets --force --delete-emptydir-data The output is similar to this:node "kubernetes-node-i4c4" cordoned WARNING: Deleting pods not managed by ReplicationController, ReplicaSet, Job, or DaemonSet: fluentd-cloud-logging-kubernetes-node-i4c4, kube-proxy-kubernetes-node-i4c4; Ignoring DaemonSet-managed pods: node-problem-detector-v0.1-dyrog WARNING: Ignoring DaemonSet-managed pods: node-problem-detector-v0.1-dyrog; Deleting pods not managed by ReplicationController, ReplicaSet, Job, or DaemonSet: fluentd-cloud-logging-kubernetes-node-i4c4, kube-proxy-kubernetes-node-i4c4 There are pending pods when an error occurred: Cannot evict pod as it would violate the pod's disruption budget. pod/zk-2 Use CTRL-C to terminate kubectl.You cannot drain the third node because evicting zk-2 would violate zk-budget. However, the node will remain cordoned.Use zkCli.sh to retrieve the value you entered during the sanity test from zk-0.kubectl exec zk-0 zkCli.sh get /hello The service is still available because its PodDisruptionBudget is respected.WatchedEvent state:SyncConnected type:None path:null world cZxid = 0x200000002 ctime = Wed Dec 07 00:08:59 UTC 2016 mZxid = 0x200000002 mtime = Wed Dec 07 00:08:59 UTC 2016 pZxid = 0x200000002 cversion = 0 dataVersion = 0 aclVersion = 0 ephemeralOwner = 0x0 dataLength = 5 numChildren = 0 Use kubectl uncordon to uncordon the first node.kubectl uncordon kubernetes-node-pb41 The output is similar to this:node "kubernetes-node-pb41" uncordoned zk-1 is rescheduled on this node. Wait until zk-1 is Running and Ready.kubectl get pods -w -l app=zk The output is similar to this:NAME READY STATUS RESTARTS AGE zk-0 1/1 Running 2 1h zk-1 1/1 Running 0 1h zk-2 1/1 Running 0 1h NAME READY STATUS RESTARTS AGE zk-0 1/1 Terminating 2 2h zk-0 0/1 Terminating 2 2h zk-0 0/1 Terminating 2 2h zk-0 0/1 Terminating 2 2h zk-0 0/1 Pending 0 0s zk-0 0/1 Pending 0 0s zk-0 0/1 ContainerCreating 0 0s zk-0 0/1 Running 0 51s zk-0 1/1 Running 0 1m zk-1 1/1 Terminating 0 2h zk-1 0/1 Terminating 0 2h zk-1 0/1 Terminating 0 2h zk-1 0/1 Terminating 0 2h zk-1 0/1 Pending 0 0s zk-1 0/1 Pending 0 0s zk-1 0/1 Pending 0 12m zk-1 0/1 ContainerCreating 0 12m zk-1 0/1 Running 0 13m zk-1 1/1 Running 0 13m Attempt to drain the node on which zk-2 is scheduled.kubectl drain $(kubectl get pod zk-2 --template {{.spec.nodeName}}) --ignore-daemonsets --force --delete-emptydir-data The output is similar to this:node "kubernetes-node-i4c4" already cordoned WARNING: Deleting pods not managed by ReplicationController, ReplicaSet, Job, or DaemonSet: fluentd-cloud-logging-kubernetes-node-i4c4, kube-proxy-kubernetes-node-i4c4; Ignoring DaemonSet-managed pods: node-problem-detector-v0.1-dyrog pod "heapster-v1.2.0-2604621511-wht1r" deleted pod "zk-2" deleted node "kubernetes-node-i4c4" drained This time kubectl drain succeeds.Uncordon the second node to allow zk-2 to be rescheduled.kubectl uncordon kubernetes-node-ixsl The output is similar to this:node "kubernetes-node-ixsl" uncordoned You can use kubectl drain in conjunction with PodDisruptionBudgets to ensure that your services remain available during maintenance. If drain is used to cordon nodes and evict pods prior to taking the node offline for maintenance, services that express a disruption budget will have that budget respected. You should always allocate additional capacity for critical services so that their Pods can be immediately rescheduled.Cleaning upUse kubectl uncordon to uncordon all the nodes in your cluster.You must delete the persistent storage media for the PersistentVolumes used in this tutorial. Follow the necessary steps, based on your environment, storage configuration, and provisioning method, to ensure that all storage is reclaimed.


# Configuration

This is the multi-page printable view of this section. Click here to print.Return to the regular view of this page.ConfigurationResources that Kubernetes provides for configuring Pods.1: Configuration Best Practices2: ConfigMaps3: Secrets4: Liveness, Readiness, and Startup Probes5: Resource Management for Pods and Containers6: Organizing Cluster Access Using kubeconfig Files7: Resource Management for Windows nodes1 - Configuration Best PracticesThis document highlights and consolidates configuration best practices that are introduced throughout the user guide, Getting Started documentation, and examples.This is a living document. If you think of something that is not on this list but might be useful to others, please don't hesitate to file an issue or submit a PR.General Configuration TipsWhen defining configurations, specify the latest stable API version.Configuration files should be stored in version control before being pushed to the cluster. This allows you to quickly roll back a configuration change if necessary. It also aids cluster re-creation and restoration.Write your configuration files using YAML rather than JSON. Though these formats can be used interchangeably in almost all scenarios, YAML tends to be more user-friendly.Group related objects into a single file whenever it makes sense. One file is often easier to manage than several. See the guestbook-all-in-one.yaml file as an example of this syntax.Note also that many kubectl commands can be called on a directory. For example, you can call kubectl apply on a directory of config files.Don't specify default values unnecessarily: simple, minimal configuration will make errors less likely.Put object descriptions in annotations, to allow better introspection.Note:There is a breaking change introduced in the YAML 1.2 boolean values specification with respect to YAML 1.1. This is a known issue in Kubernetes. YAML 1.2 only recognizes true and false as valid booleans, while YAML 1.1 also accepts yes, no, on, and off as booleans. However, Kubernetes uses YAML parsers that are mostly compatible with YAML 1.1, which means that using yes or no instead of true or false in a YAML manifest may cause unexpected errors or behaviors. To avoid this issue, it is recommended to always use true or false for boolean values in YAML manifests, and to quote any strings that may be confused with booleans, such as "yes" or "no".Besides booleans, there are additional specifications changes between YAML versions. Please refer to the YAML Specification Changes documentation for a comprehensive list."Naked" Pods versus ReplicaSets, Deployments, and JobsDon't use naked Pods (that is, Pods not bound to a ReplicaSet or Deployment) if you can avoid it. Naked Pods will not be rescheduled in the event of a node failure.A Deployment, which both creates a ReplicaSet to ensure that the desired number of Pods is always available, and specifies a strategy to replace Pods (such as RollingUpdate), is almost always preferable to creating Pods directly, except for some explicit restartPolicy: Never scenarios. A Job may also be appropriate.ServicesCreate a Service before its corresponding backend workloads (Deployments or ReplicaSets), and before any workloads that need to access it. When Kubernetes starts a container, it provides environment variables pointing to all the Services which were running when the container was started. For example, if a Service named foo exists, all containers will get the following variables in their initial environment:FOO_SERVICE_HOST=<the host the Service is running on> FOO_SERVICE_PORT=<the port the Service is running on> This does imply an ordering requirement - any Service that a Pod wants to access must be created before the Pod itself, or else the environment variables will not be populated. DNS does not have this restriction.An optional (though strongly recommended) cluster add-on is a DNS server. The DNS server watches the Kubernetes API for new Services and creates a set of DNS records for each. If DNS has been enabled throughout the cluster then all Pods should be able to do name resolution of Services automatically.Don't specify a hostPort for a Pod unless it is absolutely necessary. When you bind a Pod to a hostPort, it limits the number of places the Pod can be scheduled, because each <hostIP, hostPort, protocol> combination must be unique. If you don't specify the hostIP and protocol explicitly, Kubernetes will use 0.0.0.0 as the default hostIP and TCP as the default protocol.If you only need access to the port for debugging purposes, you can use the apiserver proxy or kubectl port-forward.If you explicitly need to expose a Pod's port on the node, consider using a NodePort Service before resorting to hostPort.Avoid using hostNetwork, for the same reasons as hostPort.Use headless Services (which have a ClusterIP of None) for service discovery when you don't need kube-proxy load balancing.Using LabelsDefine and use labels that identify semantic attributes of your application or Deployment, such as { app.kubernetes.io/name: MyApp, tier: frontend, phase: test, deployment: v3 }. You can use these labels to select the appropriate Pods for other resources; for example, a Service that selects all tier: frontend Pods, or all phase: test components of app.kubernetes.io/name: MyApp. See the guestbook app for examples of this approach.A Service can be made to span multiple Deployments by omitting release-specific labels from its selector. When you need to update a running service without downtime, use a Deployment.A desired state of an object is described by a Deployment, and if changes to that spec are applied, the deployment controller changes the actual state to the desired state at a controlled rate.Use the Kubernetes common labels for common use cases. These standardized labels enrich the metadata in a way that allows tools, including kubectl and dashboard, to work in an interoperable way.You can manipulate labels for debugging. Because Kubernetes controllers (such as ReplicaSet) and Services match to Pods using selector labels, removing the relevant labels from a Pod will stop it from being considered by a controller or from being served traffic by a Service. If you remove the labels of an existing Pod, its controller will create a new Pod to take its place. This is a useful way to debug a previously "live" Pod in a "quarantine" environment. To interactively remove or add labels, use kubectl label.Using kubectlUse kubectl apply -f <directory>. This looks for Kubernetes configuration in all .yaml, .yml, and .json files in <directory> and passes it to apply.Use label selectors for get and delete operations instead of specific object names. See the sections on label selectors and using labels effectively.Use kubectl create deployment and kubectl expose to quickly create single-container Deployments and Services. See Use a Service to Access an Application in a Cluster for an example.2 - ConfigMapsA ConfigMap is an API object used to store non-confidential data in key-value pairs. Pods can consume ConfigMaps as environment variables, command-line arguments, or as configuration files in a volume.A ConfigMap allows you to decouple environment-specific configuration from your container images, so that your applications are easily portable.Caution:ConfigMap does not provide secrecy or encryption. If the data you want to store are confidential, use a Secret rather than a ConfigMap, or use additional (third party) tools to keep your data private.MotivationUse a ConfigMap for setting configuration data separately from application code.For example, imagine that you are developing an application that you can run on your own computer (for development) and in the cloud (to handle real traffic). You write the code to look in an environment variable named DATABASE_HOST. Locally, you set that variable to localhost. In the cloud, you set it to refer to a Kubernetes Service that exposes the database component to your cluster. This lets you fetch a container image running in the cloud and debug the exact same code locally if needed.Note:A ConfigMap is not designed to hold large chunks of data. The data stored in a ConfigMap cannot exceed 1 MiB. If you need to store settings that are larger than this limit, you may want to consider mounting a volume or use a separate database or file service.ConfigMap objectA ConfigMap is an API object that lets you store configuration for other objects to use. Unlike most Kubernetes objects that have a spec, a ConfigMap has data and binaryData fields. These fields accept key-value pairs as their values. Both the data field and the binaryData are optional. The data field is designed to contain UTF-8 strings while the binaryData field is designed to contain binary data as base64-encoded strings.The name of a ConfigMap must be a valid DNS subdomain name.Each key under the data or the binaryData field must consist of alphanumeric characters, -, _ or .. The keys stored in data must not overlap with the keys in the binaryData field.Starting from v1.19, you can add an immutable field to a ConfigMap definition to create an immutable ConfigMap.ConfigMaps and PodsYou can write a Pod spec that refers to a ConfigMap and configures the container(s) in that Pod based on the data in the ConfigMap. The Pod and the ConfigMap must be in the same namespace.Note:The spec of a static Pod cannot refer to a ConfigMap or any other API objects.Here's an example ConfigMap that has some keys with single values, and other keys where the value looks like a fragment of a configuration format.apiVersion: v1 kind: ConfigMap metadata: name: game-demo data: # property-like keys; each key maps to a simple value player_initial_lives: "3" ui_properties_file_name: "user-interface.properties" # file-like keys game.properties: | enemy.types=aliens,monsters player.maximum-lives=5 user-interface.properties: | color.good=purple color.bad=yellow allow.textmode=true There are four different ways that you can use a ConfigMap to configure a container inside a Pod:Inside a container command and argsEnvironment variables for a containerAdd a file in read-only volume, for the application to readWrite code to run inside the Pod that uses the Kubernetes API to read a ConfigMapThese different methods lend themselves to different ways of modeling the data being consumed. For the first three methods, the kubelet uses the data from the ConfigMap when it launches container(s) for a Pod.The fourth method means you have to write code to read the ConfigMap and its data. However, because you're using the Kubernetes API directly, your application can subscribe to get updates whenever the ConfigMap changes, and react when that happens. By accessing the Kubernetes API directly, this technique also lets you access a ConfigMap in a different namespace.Here's an example Pod that uses values from game-demo to configure a Pod:configmap/configure-pod.yaml apiVersion: v1 kind: Pod metadata: name: configmap-demo-pod spec: containers: - name: demo image: alpine command: ["sleep", "3600"] env: # Define the environment variable - name: PLAYER_INITIAL_LIVES # Notice that the case is different here # from the key name in the ConfigMap. valueFrom: configMapKeyRef: name: game-demo # The ConfigMap this value comes from. key: player_initial_lives # The key to fetch. - name: UI_PROPERTIES_FILE_NAME valueFrom: configMapKeyRef: name: game-demo key: ui_properties_file_name volumeMounts: - name: config mountPath: "/config" readOnly: true volumes: # You set volumes at the Pod level, then mount them into containers inside that Pod - name: config configMap: # Provide the name of the ConfigMap you want to mount. name: game-demo # An array of keys from the ConfigMap to create as files items: - key: "game.properties" path: "game.properties" - key: "user-interface.properties" path: "user-interface.properties" A ConfigMap doesn't differentiate between single line property values and multi-line file-like values. What matters is how Pods and other objects consume those values.For this example, defining a volume and mounting it inside the demo container as /config creates two files, /config/game.properties and /config/user-interface.properties, even though there are four keys in the ConfigMap. This is because the Pod definition specifies an items array in the volumes section. If you omit the items array entirely, every key in the ConfigMap becomes a file with the same name as the key, and you get 4 files.Using ConfigMapsConfigMaps can be mounted as data volumes. ConfigMaps can also be used by other parts of the system, without being directly exposed to the Pod. For example, ConfigMaps can hold data that other parts of the system should use for configuration.The most common way to use ConfigMaps is to configure settings for containers running in a Pod in the same namespace. You can also use a ConfigMap separately.For example, you might encounter addons or operators that adjust their behavior based on a ConfigMap.Using ConfigMaps as files from a PodTo consume a ConfigMap in a volume in a Pod:Create a ConfigMap or use an existing one. Multiple Pods can reference the same ConfigMap.Modify your Pod definition to add a volume under .spec.volumes[]. Name the volume anything, and have a .spec.volumes[].configMap.name field set to reference your ConfigMap object.Add a .spec.containers[].volumeMounts[] to each container that needs the ConfigMap. Specify .spec.containers[].volumeMounts[].readOnly = true and .spec.containers[].volumeMounts[].mountPath to an unused directory name where you would like the ConfigMap to appear.Modify your image or command line so that the program looks for files in that directory. Each key in the ConfigMap data map becomes the filename under mountPath.This is an example of a Pod that mounts a ConfigMap in a volume:apiVersion: v1 kind: Pod metadata: name: mypod spec: containers: - name: mypod image: redis volumeMounts: - name: foo mountPath: "/etc/foo" readOnly: true volumes: - name: foo configMap: name: myconfigmap Each ConfigMap you want to use needs to be referred to in .spec.volumes.If there are multiple containers in the Pod, then each container needs its own volumeMounts block, but only one .spec.volumes is needed per ConfigMap.Mounted ConfigMaps are updated automaticallyWhen a ConfigMap currently consumed in a volume is updated, projected keys are eventually updated as well. The kubelet checks whether the mounted ConfigMap is fresh on every periodic sync. However, the kubelet uses its local cache for getting the current value of the ConfigMap. The type of the cache is configurable using the configMapAndSecretChangeDetectionStrategy field in the KubeletConfiguration struct. A ConfigMap can be either propagated by watch (default), ttl-based, or by redirecting all requests directly to the API server. As a result, the total delay from the moment when the ConfigMap is updated to the moment when new keys are projected to the Pod can be as long as the kubelet sync period + cache propagation delay, where the cache propagation delay depends on the chosen cache type (it equals to watch propagation delay, ttl of cache, or zero correspondingly).ConfigMaps consumed as environment variables are not updated automatically and require a pod restart.Note:A container using a ConfigMap as a subPath volume mount will not receive ConfigMap updates.Using Configmaps as environment variablesTo use a Configmap in an environment variable in a Pod:For each container in your Pod specification, add an environment variable for each Configmap key that you want to use to the env[].valueFrom.configMapKeyRef field.Modify your image and/or command line so that the program looks for values in the specified environment variables.This is an example of defining a ConfigMap as a pod environment variable:The following ConfigMap (myconfigmap.yaml) stores two properties: username and access_level:apiVersion: v1 kind: ConfigMap metadata: name: myconfigmap data: username: k8s-admin access_level: "1" The following command will create the ConfigMap object:kubectl apply -f myconfigmap.yaml The following Pod consumes the content of the ConfigMap as environment variables:configmap/env-configmap.yaml apiVersion: v1 kind: Pod metadata: name: env-configmap spec: containers: - name: app command: ["/bin/sh", "-c", "printenv"] image: busybox:latest envFrom: - configMapRef: name: myconfigmap The envFrom field instructs Kubernetes to create environment variables from the sources nested within it. The inner configMapRef refers to a ConfigMap by its name and selects all its key-value pairs. Add the Pod to your cluster, then retrieve its logs to see the output from the printenv command. This should confirm that the two key-value pairs from the ConfigMap have been set as environment variables:kubectl apply -f env-configmap.yaml kubectl logs pod/ env-configmap The output is similar to this:... username: "k8s-admin" access_level: "1" ... Sometimes a Pod won't require access to all the values in a ConfigMap. For example, you could have another Pod which only uses the username value from the ConfigMap. For this use case, you can use the env.valueFrom syntax instead, which lets you select individual keys in a ConfigMap. The name of the environment variable can also be different from the key within the ConfigMap. For example:apiVersion: v1 kind: Pod metadata: name: env-configmap spec: containers: - name: envars-test-container image: nginx env: - name: CONFIGMAP_USERNAME valueFrom: configMapKeyRef: name: myconfigmap key: username In the Pod created from this manifest, you will see that the environment variable CONFIGMAP_USERNAME is set to the value of the username value from the ConfigMap. Other keys from the ConfigMap data are not copied into the environment.It's important to note that the range of characters allowed for environment variable names in pods is restricted. If any keys do not meet the rules, those keys are not made available to your container, though the Pod is allowed to start.Immutable ConfigMapsFEATURE STATE: Kubernetes v1.21 [stable]The Kubernetes feature Immutable Secrets and ConfigMaps provides an option to set individual Secrets and ConfigMaps as immutable. For clusters that extensively use ConfigMaps (at least tens of thousands of unique ConfigMap to Pod mounts), preventing changes to their data has the following advantages:protects you from accidental (or unwanted) updates that could cause applications outagesimproves performance of your cluster by significantly reducing load on kube-apiserver, by closing watches for ConfigMaps marked as immutable.You can create an immutable ConfigMap by setting the immutable field to true. For example:apiVersion: v1 kind: ConfigMap metadata: ... data: ... immutable: true Once a ConfigMap is marked as immutable, it is not possible to revert this change nor to mutate the contents of the data or the binaryData field. You can only delete and recreate the ConfigMap. Because existing Pods maintain a mount point to the deleted ConfigMap, it is recommended to recreate these pods.What's


# kubectl for Docker Users

kubectl for Docker UsersYou can use the Kubernetes command line tool kubectl to interact with the API Server. Using kubectl is straightforward if you are familiar with the Docker command line tool. However, there are a few differences between the Docker commands and the kubectl commands. The following sections show a Docker sub-command and describe the equivalent kubectl command.docker runTo run an nginx Deployment and expose the Deployment, see kubectl create deployment.docker:docker run -d --restart=always -e DOMAIN=cluster --name nginx-app -p 80:80 nginx 55c103fa129692154a7652490236fee9be47d70a8dd562281ae7d2f9a339a6db docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 55c103fa1296 nginx "nginx -g 'daemon of…" 9 seconds ago Up 9 seconds 0.0.0.0:80->80/tcp nginx-app kubectl:# start the pod running nginx kubectl create deployment --image=nginx nginx-app deployment.apps/nginx-app created # add env to nginx-app kubectl set env deployment/nginx-app DOMAIN=cluster deployment.apps/nginx-app env updated Note:kubectl commands print the type and name of the resource created or mutated, which can then be used in subsequent commands. You can expose a new Service after a Deployment is created.# expose a port through with a service kubectl expose deployment nginx-app --port=80 --name=nginx-http service "nginx-http" exposed By using kubectl, you can create a Deployment to ensure that N pods are running nginx, where N is the number of replicas stated in the spec and defaults to 1. You can also create a service with a selector that matches the pod labels. For more information, see Use a Service to Access an Application in a Cluster.By default images run in the background, similar to docker run -d .... To run things in the foreground, use kubectl run to create pod:kubectl run [-i] [--tty] --attach <name> --image=<image> Unlike docker run ..., if you specify --attach, then you attach stdin, stdout and stderr. You cannot control which streams are attached (docker -a ...). To detach from the container, you can type the escape sequence Ctrl+P followed by Ctrl+Q.docker psTo list what is currently running, see kubectl get.docker:docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 14636241935f ubuntu:16.04 "echo test" 5 seconds ago Exited (0) 5 seconds ago cocky_fermi 55c103fa1296 nginx "nginx -g 'daemon of…" About a minute ago Up About a minute 0.0.0.0:80->80/tcp nginx-app kubectl:kubectl get po NAME READY STATUS RESTARTS AGE nginx-app-8df569cb7-4gd89 1/1 Running 0 3m ubuntu 0/1 Completed 0 20s docker attachTo attach a process that is already running in a container, see kubectl attach.docker:docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 55c103fa1296 nginx "nginx -g 'daemon of…" 5 minutes ago Up 5 minutes 0.0.0.0:80->80/tcp nginx-app docker attach 55c103fa1296 ... kubectl:kubectl get pods NAME READY STATUS RESTARTS AGE nginx-app-5jyvm 1/1 Running 0 10m kubectl attach -it nginx-app-5jyvm ... To detach from the container, you can type the escape sequence Ctrl+P followed by Ctrl+Q.docker execTo execute a command in a container, see kubectl exec.docker:docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 55c103fa1296 nginx "nginx -g 'daemon of…" 6 minutes ago Up 6 minutes 0.0.0.0:80->80/tcp nginx-app docker exec 55c103fa1296 cat /etc/hostname 55c103fa1296 kubectl:kubectl get po NAME READY STATUS RESTARTS AGE nginx-app-5jyvm 1/1 Running 0 10m kubectl exec nginx-app-5jyvm -- cat /etc/hostname nginx-app-5jyvm To use interactive commands.docker:docker exec -ti 55c103fa1296 /bin/sh # exit kubectl:kubectl exec -ti nginx-app-5jyvm -- /bin/sh # exit For more information, see Get a Shell to a Running Container.docker logsTo follow stdout/stderr of a process that is running, see kubectl logs.docker:docker logs -f a9e 192.168.9.1 - - [14/Jul/2015:01:04:02 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.35.0" "-" 192.168.9.1 - - [14/Jul/2015:01:04:03 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.35.0" "-" kubectl:kubectl logs -f nginx-app-zibvs 10.240.63.110 - - [14/Jul/2015:01:09:01 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.26.0" "-" 10.240.63.110 - - [14/Jul/2015:01:09:02 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.26.0" "-" There is a slight difference between pods and containers; by default pods do not terminate if their processes exit. Instead the pods restart the process. This is similar to the docker run option --restart=always with one major difference. In docker, the output for each invocation of the process is concatenated, but for Kubernetes, each invocation is separate. To see the output from a previous run in Kubernetes, do this:kubectl logs --previous nginx-app-zibvs 10.240.63.110 - - [14/Jul/2015:01:09:01 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.26.0" "-" 10.240.63.110 - - [14/Jul/2015:01:09:02 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.26.0" "-" For more information, see Logging Architecture.docker stop and docker rmTo stop and delete a running process, see kubectl delete.docker:docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES a9ec34d98787 nginx "nginx -g 'daemon of" 22 hours ago Up 22 hours 0.0.0.0:80->80/tcp, 443/tcp nginx-app docker stop a9ec34d98787 a9ec34d98787 docker rm a9ec34d98787 a9ec34d98787 kubectl:kubectl get deployment nginx-app NAME READY UP-TO-DATE AVAILABLE AGE nginx-app 1/1 1 1 2m kubectl get po -l app=nginx-app NAME READY STATUS RESTARTS AGE nginx-app-2883164633-aklf7 1/1 Running 0 2m kubectl delete deployment nginx-app deployment "nginx-app" deleted kubectl get po -l app=nginx-app # Return nothing Note:When you use kubectl, you don't delete the pod directly. You have to first delete the Deployment that owns the pod. If you delete the pod directly, the Deployment recreates the pod.docker loginThere is no direct analog of docker login in kubectl. If you are interested in using Kubernetes with a private registry, see Using a Private Registry.docker versionTo get the version of client and server, see kubectl version.docker:docker version Client version: 1.7.0 Client API version: 1.19 Go version (client): go1.4.2 Git commit (client): 0baf609 OS/Arch (client): linux/amd64 Server version: 1.7.0 Server API version: 1.19 Go version (server): go1.4.2 Git commit (server): 0baf609 OS/Arch (server): linux/amd64 kubectl:kubectl version Client Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.9+a3d1dfa6f4335", GitCommit:"9b77fed11a9843ce3780f70dd251e92901c43072", GitTreeState:"dirty", BuildDate:"2017-08-29T20:32:58Z", OpenPaasKubernetesVersion:"v1.03.02", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.9+a3d1dfa6f4335", GitCommit:"9b77fed11a9843ce3780f70dd251e92901c43072", GitTreeState:"dirty", BuildDate:"2017-08-29T20:32:58Z", OpenPaasKubernetesVersion:"v1.03.02", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"} docker infoTo get miscellaneous information about the environment and configuration, see kubectl cluster-info.docker:docker info Containers: 40 Images: 168 Storage Driver: aufs Root Dir: /usr/local/google/docker/aufs Backing Filesystem: extfs Dirs: 248 Dirperm1 Supported: false Execution Driver: native-0.2 Logging Driver: json-file Kernel Version: 3.13.0-53-generic Operating System: Ubuntu 14.04.2 LTS CPUs: 12 Total Memory: 31.32 GiB Name: k8s-is-fun.mtv.corp.google.com ID: ADUV:GCYR:B3VJ:HMPO:LNPQ:KD5S:YKFQ:76VN:IANZ:7TFV:ZBF4:BYJO WARNING: No swap limit support kubectl:kubectl cluster-info Kubernetes master is running at https://203.0.113.141 KubeDNS is running at https://203.0.113.141/api/v1/namespaces/kube-system/services/kube-dns/proxy kubernetes-dashboard is running at https://203.0.113.141/api/v1/namespaces/kube-system/services/kubernetes-dashboard/proxy Grafana is running at https://203.0.113.141/api/v1/namespaces/kube-system/services/monitoring-grafana/proxy Heapster is running at https://203.0.113.141/api/v1/namespaces/kube-system/services/monitoring-heapster/proxy InfluxDB is running at https://203.0.113.141/api/v1/namespaces/kube-system/services/monitoring-influxdb/proxy