The awfulness of AWS EKS

  • what EKS calls setting up a cluster merely sets up the control plane / master. Only then are you directed to separately setup node pools and the like.
  • Of course, EKS doesn’t create all the networking for you … oh no. Where the other providers allow you to setup the structure if you don’t have it, AWS doesn’t. You’d better setup everything right or stuff just won’t work and you have no idea why
  • At launch, EKS was clearly the unwanted stepchild: it wasn’t offered a first class citizen place in aws cli tools (in the way GKE did). Took almost 4 years to be able to do some stuff in aws cli tools, until AWS bought eksctl … which is still worlds apart from being the declarative tool they’ve been advertising.
  • when you try to discuss EKS with AWS support, they don’t really focus on the Kubernetes platform, but rather on the rest of AWS concepts and EKS platform seems foreign to them (“why did you setup the load balancer target group like that, you should have only 2 instances there” — “I didn’t set it up, that’s how the ingress interacts with the platform, it registers all instances regardless whether ingress pods run there …. “why is the autoscaling group changing by itself” — “because it’s managed by the autoscaler”)
  • you need to make sure all proper policies are in places and assigned (including, for example CNI policies though weirdly AWS only provides an aws-managed policy for ipv4 whereas for ipv6 cluster you need to create your own)
  • cluster autoscaler isn’t even a plugin (though weirdly things like ebs csi controller and vpc cni are considered plugins, though they are enabled by default)
  • you get instructions on how to remove pod count limit, by editing launch templates instead of providing launch templates with the limit removed, even in cases where the limit shouldn’t exist (like for ipv6 clusters)
  • the whole ridiculous pod count per node limit only exists due to the insistence of tying the internal cluster networking to the VPC. Sure, it’s great if you plan to route traffic directly to services/pods and it’s sure good to have the option, but it shouldn’t be the default — it should be an option for which the users know the limitation of and should choose whether to use it. In other words, cni overlay should be the default and vpc cni should be an option.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store