Home • Explore • Blog • CI
☰



kubernetes
2nd chunk of `content/en/blog/_posts/2017-08-00-High-Performance-Networking-With-Ec2.md`
4fde2a9f49fb46913452f63de67c1d74dc889eb799a1319200000001000008e1



## Kubernetes on Multi-Segment Networks



The way to avoid running out of VPC routes is to use them sparingly by making them forward pod traffic for multiple instances. From a networking perspective, what that means is that the VPC route needs to forward to a router, which can then forward traffic on to the final destination instance.


Romana is a CNI network provider that configures routes on the host to forward pod network traffic without an overlay. Since inter-node routes are installed on hosts, no VPC routes are necessary at all. However, when the VPC is split into subnets for an HA deployment across zones, VPC routes are necessary.


Fortunately, inter-node routes on hosts allows them to act as a network router and forward traffic inbound from another zone just as it would for traffic from local pods. This makes any Kubernetes node configured by Romana able to accept inbound pod traffic from other zones and forward it to the proper destination node on the subnet.


Because of this local routing function, top-level routes to pods on other instances on the subnet can be aggregated, collapsing the total number of routes necessary to as few as one per subnet. To avoid using a single instance to forward all traffic, more routes can be used to spread traffic across multiple instances, up to the maximum number of available routes (i.e. equivalent to kubenet).


The net result is that you can now build clusters of any size across AZs without an overlay. Romana clusters also support network policies for better security through network isolation.


## Making it All Work


While the combination of aggregated routes and node forwarding on a subnet eliminates overlays and avoids the VPC 50 route limitation, it imposes certain requirements on the CNI provider. For example, hosts should be configured with inter-node routes only to other nodes in the same zone on the local subnet. Traffic to all other hosts must use the default route off host, then use the (aggregated) VPC route to forward traffic out of the zone. Also: when adding a new host, in order to maintain aggregated VPC routes, the CNI plugin needs to use IP addresses for pods that are reachable on the new host.
Title: Kubernetes on Multi-Segment Networks
Summary
To avoid VPC route limits, Romana uses host-based routing to forward pod traffic without overlays. Nodes act as routers, enabling traffic forwarding across zones. This allows route aggregation, reducing the number of VPC routes needed. Romana supports large clusters across AZs with network policies. The CNI provider must configure inter-node routes only to nodes in the same zone, using the default route for traffic to other zones. IP addresses for pods must be reachable on new hosts to maintain aggregated VPC routes.