Kafka Cluster Deployment, Data Production/Consumption, and Updates

Note: Details of the manifests contents can be found [here](https://kow3ns.github.io/kubernetes-kafka/manifests/). You can learn more about running a Kafka cluster on Kubernetes [here](https://kow3ns.github.io/kubernetes-kafka/). To create a cluster, you only need to download and apply the [kafka\_mini.yaml](https://kow3ns.github.io/kubernetes-kafka/manifests/kafka_mini.yaml) manifest. When you apply the manifest, you will see output like the following: ``` $ kubectl apply -f kafka\_mini.yaml service "kafka-hs" created poddisruptionbudget "kafka-pdb" created statefulset "kafka" created ``` The manifest creates a three broker cluster using the kafka StatefulSet, a Headless Service, kafka-hs, to control the domain of the brokers; and a PodDisruptionBudget, kafka-pdb, that allows for one planned disruption. The brokers are configured to use the ZooKeeper ensemble we created above by connecting through the zk-cs Service. As with the ZooKeeper ensemble deployed above, this Kafka cluster is fine for demonstration purposes, but it’s probably not sized correctly for production use. If you watch Pod creation, you will notice that, like the ZooKeeper ensemble created above, the Kafka cluster uses the Parallel podManagementPolicy. ``` $ kubectl get po -lapp=kafka -w NAME READY STATUS RESTARTS AGE kafka-0 0/1 Pending 0 0s kafka-0 0/1 Pending 0 0s kafka-1 0/1 Pending 0 0s kafka-1 0/1 Pending 0 0s kafka-2 0/1 Pending 0 0s kafka-0 0/1 ContainerCreating 0 0s kafka-2 0/1 Pending 0 0s kafka-1 0/1 ContainerCreating 0 0s kafka-1 0/1 Running 0 11s kafka-0 0/1 Running 0 19s kafka-1 1/1 Running 0 23s kafka-0 1/1 Running 0 32s ``` ## Producing and consuming data You can use kubectl run to execute the kafka-topics.sh script to create a topic named test. ``` $ kubectl run -ti --image=gcr.io/google\_containers/kubernetes-kafka:1.0-10.2.1 createtopic --restart=Never --rm -- kafka-topics.sh --create \ \> --topic test \ \> --zookeeper zk-cs.default.svc.cluster.local:2181 \ \> --partitions 1 \ \> --replication-factor 3 ``` Now you can use kubectl run to execute the kafka-console-consumer.sh command to listen for messages. ``` $ kubectl run -ti --image=gcr.io/google\_containers/kubnetes-kafka:1.0-10.2.1 consume --restart=Never --rm -- kafka-console-consumer.sh --topic test --bootstrap-server kafka-0.kafka-hs.default.svc.cluster.local:9093 ``` In another terminal, you can run the kafka-console-producer.sh command. ``` $kubectl run -ti --image=gcr.io/google\_containers/kubernetes-kafka:1.0-10.2.1 produce --restart=Never --rm \ \> -- kafka-console-producer.sh --topic test --broker-list kafka-0.kafka-hs.default.svc.cluster.local:9093,kafka-1.kafka-hs.default.svc.cluster.local:9093,kafka-2.kafka-hs.default.svc.cluster.local:9093 ``` Output from the second terminal appears in the first terminal. If you continue to produce and consume messages while updating the cluster, you will notice that no messages are lost. You may see error messages as the leader for the partition changes when individual brokers are updated, but the client retries until the message is committed. This is due to the ordered, sequential nature of StatefulSet rolling updates which we will explore further in the next section. Updating the Kafka cluster StatefulSet updates are like DaemonSet updates in that they are both configured by setting the spec.updateStrategy of the corresponding API object. When the update strategy is set to OnDelete, the respective controllers will only create new Pods when a Pod in the StatefulSet or DaemonSet has been deleted. When the update strategy is set to RollingUpdate, the controllers will delete and recreate Pods when a modification is made to the spec.template field of a DaemonSet or StatefulSet. You can use rolling updates to change the configuration (via environment variables or command line parameters), resource requests, resource limits, container images, labels, and/or annotations of the Pods in a StatefulSet or DaemonSet. Note that all updates are destructive, always requiring that each Pod in the DaemonSet or StatefulSet be destroyed and recreated. StatefulSet rolling updates differ from DaemonSet rolling updates in that Pod termination and creation is ordered and sequential.

This section details deploying a Kafka cluster using a manifest, which creates a three-broker setup with a Headless Service and PodDisruptionBudget, employing a Parallel podManagementPolicy for faster Pod creation. It then outlines producing and consuming data using `kafka-topics.sh`, `kafka-console-consumer.sh`, and `kafka-console-producer.sh`, ensuring no message loss during cluster updates due to client retries. Finally, it describes StatefulSet updates, configured via `spec.updateStrategy`, and how RollingUpdate destructively modifies Pods in an ordered, sequential manner, differing from DaemonSet updates.