Challenges and Solutions in PMK: Containerized Kubelet and Networking

![architecture.png](https://lh6.googleusercontent.com/ZQZoFL6tDpkiberG_X1CREitwNIDCHnRajnOlJqByU-4HzRQi1RRoDlGj7pGRaqD2a7Yg4xBwQx7oHp_mR8ie96O5w_KMT84av-JMsPMHXeoBpVYn3iJKeGZkWG4q0J06OZMuLIe) **Containerized kubelet?** Another hurdle we encountered resulted from our original decision to run kubelet as recommended by the [Multi-node Docker Deployment Guide](/docs/getting-started-guides/docker-multinode/). We discovered that this approach introduces complexities that led to many difficult-to-troubleshoot bugs that were sensitive to the combined versions of Kubernetes, Docker, and the node OS. Example: kubelet’s need to mount directories containing secrets into containers to support the [Service Accounts](/docs/user-guide/service-accounts/) mechanism. It turns out that [doing this from inside of a container is tricky](https://github.com/kubernetes/kubernetes/issues/6848), and requires a [complex sequence of steps](https://github.com/kubernetes/kubernetes/blob/release-1.0/pkg/util/mount/nsenter_mount.go#L37) that turned out to be fragile. After fixing a continuing stream of issues, we finally decided to run kubelet as a native program on the host OS, resulting in significantly better stability. **Overcoming networking hurdles** The Beta release of PMK currently uses [flannel with UDP back-end](https://github.com/coreos/flannel) for the network layer. In a Kubernetes cluster, many infrastructure services need to communicate across nodes using a variety of ports (443, 4001, etc..) and protocols (TCP and UDP). Often, customer nodes intentionally or unintentionally block some or all of the traffic, or run existing services that conflict with the required ports, resulting in non-obvious failures. To address this, we try to detect configuration problems early and inform the administrator immediately. PMK runs a “preflight” check on all nodes participating in a cluster before installing the Kubernetes software. This means running small test programs on each node to verify that (1) the required ports are available for binding and listening; and (2) nodes can connect to each other using all required ports and protocols. These checks run in parallel and take less than a couple of seconds before cluster initialization.

PMK initially ran kubelet as recommended by the Multi-node Docker Deployment Guide, but this introduced complexities and bugs related to mounting directories with secrets into containers. The solution was to run kubelet as a native program on the host OS for better stability. For networking, PMK uses flannel with a UDP back-end and addresses potential communication issues by running a 'preflight' check on all nodes before installing Kubernetes. This check verifies that required ports are available and that nodes can connect to each other using necessary ports and protocols, detecting configuration problems early.