Deploying ZooKeeper and Kafka Clusters with Parallel Pod Management

When you apply the manifest, you will see output like the following. ``` $ kubectl apply -f zookeeper\_mini.yaml service "zk-hs" created service "zk-cs" created poddisruptionbudget "zk-pdb" created statefulset "zk" created ``` The manifest creates an ensemble of three ZooKeeper servers using a StatefulSet, zk; a Headless Service, zk-hs, to control the domain of the ensemble; a Service, zk-cs, that clients can use to connect to the ready ZooKeeper instances; and a PodDisruptionBugdet, zk-pdb, that allows for one planned disruption. (Note that while this ensemble is suitable for demonstration purposes, it isn’t sized correctly for production use.) If you use kubectl get to watch Pod creation in another terminal you will see that, in contrast to the [OrderedReady](/docs/concepts/workloads/controllers/statefulset/#orderedready-pod-management) strategy (the default policy that implements the full version of the StatefulSet guarantees), all of the Pods in the zk StatefulSet are created in parallel. ``` $ kubectl get po -lapp=zk -w NAME READY STATUS RESTARTS AGE zk-0 0/1 Pending 0 0s zk-0 0/1 Pending 0 0s zk-1 0/1 Pending 0 0s zk-1 0/1 Pending 0 0s zk-0 0/1 ContainerCreating 0 0s zk-2 0/1 Pending 0 0s zk-1 0/1 ContainerCreating 0 0s zk-2 0/1 Pending 0 0s zk-2 0/1 ContainerCreating 0 0s zk-0 0/1 Running 0 10s zk-2 0/1 Running 0 11s zk-1 0/1 Running 0 19s zk-0 1/1 Running 0 20s zk-1 1/1 Running 0 30s zk-2 1/1 Running 0 30s ``` This is because the zookeeper\_mini.yaml manifest sets the podManagementPolicy of the StatefulSet to Parallel. ``` apiVersion: apps/v1beta1 kind: StatefulSet metadata: name: zk spec: serviceName: zk-hs replicas: 3 updateStrategy: type: RollingUpdate podManagementPolicy: Parallel ... ``` Many distributed systems, like ZooKeeper, do not require ordered creation and termination for their processes. You can use the Parallel Pod management policy to accelerate the creation and deletion of StatefulSets that manage these systems. Note that, when Parallel Pod management is used, the StatefulSet controller will not block when it fails to create a Pod. Ordered, sequential Pod creation and termination is performed when a StatefulSet’s podManagementPolicy is set to OrderedReady. ## Deploying a Kafka Cluster Apache Kafka is a popular distributed streaming platform. Kafka producers write data to partitioned topics which are stored, with a configurable replication factor, on a cluster of brokers. Consumers consume the produced data from the partitions stored on the brokers. Note: Details of the manifests contents can be found [here](https://kow3ns.github.io/kubernetes-kafka/manifests/). You can learn more about running a Kafka cluster on Kubernetes [here](https://kow3ns.github.io/kubernetes-kafka/). To create a cluster, you only need to download and apply the [kafka\_mini.yaml](https://kow3ns.github.io/kubernetes-kafka/manifests/kafka_mini.yaml) manifest. When you apply the manifest, you will see output like the following: ``` $ kubectl apply -f kafka\_mini.yaml service "kafka-hs" created poddisruptionbudget "kafka-pdb" created statefulset "kafka" created ``` The manifest creates a three broker cluster using the kafka StatefulSet, a Headless Service, kafka-hs, to control the domain of the brokers; and a PodDisruptionBudget, kafka-pdb, that allows for one planned disruption. The brokers are configured to use the ZooKeeper ensemble we created above by connecting through the zk-cs Service. As with the ZooKeeper ensemble deployed above, this Kafka cluster is fine for demonstration purposes, but it’s probably not sized correctly for production use.

This section illustrates deploying a ZooKeeper ensemble and a Kafka cluster using StatefulSets and the 'Parallel' Pod management policy in Kubernetes. Applying the ZooKeeper manifest creates a three-server ensemble with services for domain control and client connections, while the Parallel policy allows for concurrent Pod creation, speeding up the process. Similarly, deploying the Kafka manifest creates a three-broker cluster configured to use the ZooKeeper ensemble, with services for domain control. Both deployments are suitable for demonstration but may need scaling for production use.