Kubernetes: AWS vs GCP vs Azure vs DigitalOcean

Not log ago I read a Medium article on AWS vs GCP vs Azure with respect to the ability of managing Kubernetes clusters. Since then, DigitalOcean has launched a beta of their own Kubernetes solution while AWS has pushed their EKS service out. GCP has also updated their platform.

Given the popularity of Kubernetes, I’d say it’s time to have a look at what’s what (through a beginner’s eyes — my eyes) while keeping costs in mind as well.

Disclaimer: I have some mild personal views that may affect the way I consider the comparison.

Personal views

AWS: expensive, awkward policy for rolling out updates and new policies (the zones I used most are near last to get goodies), UI still feels slow and clunky. Good documentation but still feels like you need a PhD to use aws cli tools. Technical support seems limited to providing copy/paste from docs, still useful at times (docs are not easy to navigate, easy to miss stuff).

GCP: clean UI, intuitive, lower costs and I like their sustained use discounts. They tend to keep everything at the forefront with new features and so on. I love that every step in the Console has a button to generate CLI commands, very useful for learning. Ability to customise CPU/RAM is priceless.

Azure: UI between AWS and GCP, but often options feel hidden or totally counter-intuitive. Feels expensive for what it offers.

DigitalOcean: cheap, great technical docs and support, feels made for developers but missing the managed services offered by others. Nice move with Kubernetes though.

Comparison

Let’s move on to the Kubernetes offers. For comparison, I aim at a basic 3-node cluster using 2 CPU and 4 Gb of RAM, just for comparison (storage not included).

1. Digital Ocean

Setup: you get to define a node pool by choosing how many nodes and what nodes you want (and some labels) and you’re good to go. The next step is a walkthrough on downloading kubectl configuration to access the cluster (no autoconfigure here, if you want you can use the provided file to amend your kubectl configuration manually), deploying things, using persistent volumes.

Management: as no-frills as it gets. There’s no management UI (you can deploy Kubernetes Dashboard yourself), no metrics, no monitoring. You can add/remove nodes, add/remove node pools and that’s about it.

Cost: per above configuration, 3 nodes come at $60 for the lot. Master node and control plane are free! In this configuration you also get the default storage that comes with each node type, while extra volumes cost more.

Good: you get loads of documentation, tutorials, how-to’s to deploy and manage any application (Istio, Dashboard, Prometheus, etc). It’s possibly the most complete and usable documentation, just above GCP’s. As of now, you’re getting the latest and the greatest version, 1.13 with 1.12 and 1.13 supported (only DigitalOcean offers this)

Bad: no kubectl autoconfiguration is pretty annoying if you use other systems in parallel, setting up a production-ready system takes time.

2. AWS

I can’t shake the feeling AWS came late to the Kubernetes party. EKS arrived late across Europe (late last year). For the priciest cloud out there, that’s bad. On the other hand, in most articles online you will see Kops as the recommended tool, but that’s not necessary anymore so there’s that.

Setup: You do the AWS dance (provide a name, role, subnets and security group). This gives you only the control plane in the most counter-intuitive setup I’ve seen. The next step is to go to CloudFormation and define worker nodes to add (here’s a bit of documentation).

Management: my main gripe with AWS is that they pretend to give you something and you get nothing. Stick to CloudFormation and you’re good to go, but the access setup with kubectl is counter intuitive. You get no monitoring and not much of anything.

Bad: Control plane costs $0.20 / hour. Latest Kubernetes version as of now is 1.11. EKS barely deserves it’s name, it doesn’t feel like much of a service by any means. Setup is barely a step over kops.

Good: to be discovered.

3. Google Kubernetes Engine

You can see from a mile away that in Kubernetes territory, Google is the elder, despite its shortcomings in the general cloud services area.

Setup: At the other end of the spectrum, GKE gives you options of the good kind. Setup means getting the control plane up, defining a node pool, customising instances and selecting various options. Node autoscaling, auto update Kubernetes for minor versions, Stackdriver logging & monitoring, whether to add Istio service mesh, authentication options (via IAM, basic http auth — I count this as bad, they recommend not to use it but still offer it), enable HTTP load balancing (you don’t need to, unless you want to use Google’s own ingress controller to be deployed by default), define metadata for nodes, enable secrets encryption (a beta feature in Kubernetes), deploy Kubernetes Dashboard automatically (you don’t need to since GCP offers its own dashboard with its pros/cons), node provisioning and vertical autoscaling for pods.

Management: you get a nice dashboard listing just about every Kubernetes entity (awkwardly grouped into workloads, services, storage and configuration), automatic kubectl configuration via gcloud cli, every step (including setup) has a button at the bottom of the page generating gcloud commands.

Good: lots of options and useful ones! Istio deployment, autoscaling of pods (since v1.12). Free master/control plane.

Bad: Stackdriver is the default metrics/monitoring tool and it’s pricy.

4. Azure

Setup: fairly straightforward. Select your subscription type and resource group (or create one). Select your node size and node count, authentication model, enable RBAC (not sure why this is optional as it’s the standard control tool for Kubernetes resources), http application routing (not sure why this is here — basically it allows direct access to ingresses, as if it were a bare metal setup), monitoring (nothing fancy, just enable log collection and monitoring) and that’s that. Sadly, no node pool definitions.

Management: awkward. The Kubernetes services page for a cluster is cluttered with lots of stuff that has little to do with the cluster itself. You get dedicated IAM page for something that should be a simple edit/assignment operation, you get a properties page that lists stuff that should be part of the dashboard (the dashboard, incidentally, provides no information about your cluster but seems to focus on your subscription). You also get access to Kubernetes Dashboard, installed by default, via port forwarding — there’s an azure cli command generated for you.

Good: way better than AWS, pricing is not too shabby. If you rely on terraform and azure cli for management, things go smoothly. Setup is a no-nonsense process, next to DigitalOcean’s.

Bad: clunky UI needs lots of work, setup is fairly awkward, could do with a bit more hand-holding with respect to setting up kubectl (it’s a plain azure cli command that’s not directly provided), lack of pools or any straighforward autoscaling hurts the usability in production systems.

There’s a fairly recent detailed pricing comparison available.

Google Kubernetes Engine is a clear winner, given its more that fair pricing scheme compared with lots of production-worthy features, no-nonsense UI that’s actually easy to navigate and it comes with aides to get you going with its CLI tools as well. I would add that Google’s support will go a long way towards the more technical users (same as DigitalOcean’s) while the newcomers may find it frustrating (on a ticket related to some autoscaling issues, I got queried about pod resource limits and how I got to assessing requirements for that). Also, GCP’s large network comes with lots of options and optimisations to make your application accessible with low latency. Though I have a few years with AWS, I find GCP better performing on this.

DigitalOcean: no-frills, aimed at developers. If you want a low cost managed Kubernetes solution that will hold your hand through the basics, this is it. You can develop it to production readiness for small to medium applications, as long as you keep an eye on your monitoring needs and consider that DO doesn’t have the global coverage of AWS or GCP.

Azure: it’s ok, but I have a general feeling that the pricing is not worth the offer. If you bought into Microsoft, it’s fine to stick with it but you’re missing out on lots of things that make your life easier, where autoscaling and better monitoring systems along with an actually useful dashboard are just the tip of the iceberg.

AWS: if you don’t have a choice, you should know AWS has Kubernetes, somehow, but you better get your PhD in CloudFormation. On my end, I’m not sure I would use it, even with a gun to my head.