Staging Updates and Rolling out a Canary with StatefulSets

kafka-0 0/1 Terminating 0 14m kafka-0 0/1 Terminating 0 14m kafka-0 0/1 Pending 0 0s kafka-0 0/1 Pending 0 0s kafka-0 0/1 ContainerCreating 0 0s kafka-0 0/1 Running 0 10s kafka-0 1/1 Running 0 22s ``` Note that unplanned disruptions will not lead to unintentional updates during the update process. That is, the StatefulSet controller will always recreate the Pod at the correct version to ensure the ordering of the update is preserved. If a Pod is deleted, and if it has already been updated, it will be created from the updated version of the StatefulSet’s spec.template. If the Pod has not already been updated, it will be created from the previous version of the StatefulSet’s spec.template. We will explore this further in the following sections. ## Staging an update Depending on how your organization handles deployments and configuration modifications, you may want or need to stage updates to a StatefulSet prior to allowing the roll out to progress. You can accomplish this by setting a partition for the RollingUpdate. When the StatefulSet controller detects a partition in the updateStrategy of a StatefulSet, it will only apply the updated version of the StatefulSet’s spec.template to Pods whose ordinal is greater than or equal to the value of the partition. You can patch the kafka StatefulSet to add a partition to the RollingUpdate update strategy. If you set the partition to a number greater than or equal to the StatefulSet’s spec.replicas (as below), any subsequent updates you perform to the StatefulSet’s spec.template will be staged for roll out, but the StatefulSet controller will not start a rolling update. ``` $ kubectl patch sts kafka -p '{"spec":{"updateStrategy":{"type":"RollingUpdate","rollingUpdate":{"partition":3}}}}' statefulset "kafka" patched ``` If you patch the StatefulSet to set the requested CPU to 0.3, you will notice that none of the Pods are updated. ``` $ kubectl patch sts kafka --type='json' -p='[{"op": "replace", "path": "/spec/template/spec/containers/0/resources/requests/cpu", "value":"0.3"}]' statefulset "kafka" patched ``` Even if you delete a Pod and wait for the StatefulSet controller to recreate it, you will notice that the Pod is recreated with current CPU request. ``` $ kubectl delete po kafka-1 pod "kafka-1" deleted $ kubectl get po kafka-1 -w NAME READY STATUS RESTARTS AGE kafka-1 0/1 ContainerCreating 0 10s kafka-1 0/1 Running 0 19s kafka-1 1/1 Running 0 21s $ kubectl get po kafka-1 -o yaml apiVersion: v1 kind: Pod metadata: ... resources: requests: cpu: 250m memory: 1Gi ``` ## Rolling out a canary Often, we want to verify an image update or configuration change on a single instance of an application before rolling it out globally. If you modify the partition created above to be 2, the StatefulSet controller will roll out a [canary](http://whatis.techtarget.com/definition/canary-canary-testing) that can be used to verify that the update is working as intended. ``` $ kubectl patch sts kafka -p '{"spec":{"updateStrategy":{"type":"RollingUpdate","rollingUpdate":{"partition":2}}}}' statefulset "kafka" patched ``` You can watch the StatefulSet controller update the kafka-2 Pod and pause after the update is complete. ``` $ kubectl get po -lapp=kafka -w NAME READY STATUS RESTARTS AGE kafka-0 1/1 Running 0 50m kafka-1 1/1 Running 0 10m kafka-2 1/1 Running 0 29s

This section explains how to stage updates to a StatefulSet using a partition in the RollingUpdate strategy. Setting a partition prevents updates from rolling out to Pods with ordinals less than the partition value. It also demonstrates how to roll out a canary by setting the partition to a value that updates only a specific Pod, allowing for verification before a full rollout. Examples using `kubectl patch` are provided to adjust the partition and observe the update behavior.