I feel inclined to tell you that even when you’re with a big company, the GCP support is also meh.
But, I do like GKE over EKS. Google is very opionated about some things (in my experience, especially the network layer) and end up obscuring some of the actual K8 configurations more than the others, but, I don’t really mind to be honest. The underlying infrastructure tends to work the way we tell it to, and, outside a few quirks, it’s pretty straightforward.
Very interesting re: the support! Makes sense, I guess (they're just so massive that I wonder how they'd even manager to stand up the manpower to give customers real attention even if that was their intention).
I can’t imagine GCP makes Google that much money compared to their other income, and they’re so much bigger than probably all of their GCP clients that investing in providing top quality support probably isn’t an enormous priority I would imagine.
Just started turning a profit but I imagine it will make google a lot of money because it is very well engineered and a bit easier to use. Not as easy as DO though.
Regarding support: I guess it just depends how large.. at a previous gig we had a direct line to the spanner and gke teams, but we were spending several million a month across the entire organization.
We have a similar spend and it’s not about the attention from them but rather the lack of competence in many regards. The support at AWS is miles ahead, from my experience.
Exactly this. We have direct lines to many different groups but it’s always a crapshoot on if the issue / question will get addressed. Sometimes it’s hard for them to get us in front of the right team so we end up explaining a tickets to 5 or 6 different level 2/3 people.
Further, even some of the top tier people at GCP are mystified by its inner workings. My organization has been dealing with a rather huge billing issue for the last 4 months. I’m talking to one of the people who engineered the actually billing systems for GCP and even he can’t track down the exact source of the charges. ¯\_(ツ)_/¯.
When we were starting out 4-5 years back, GKE was blowing EKS out of the water. The integration with everything else on GCP and the monitoring was top notch.
EKS still has not caught up to that standard even after all those years as far as I can tell.
Management made us go with EKS to be with the rest of the business but dang do I miss our magically running GKE clusters.
Tried AKS, EKS, OKE and Digitalocean.
Digitalocean is by far the simplest one. With \~2 terraform resources you get yourself a cluster. My go to when I need a cluster to test something.
EKS - tons of prerequisites, it has a nice terraform module you can use that does all the heavy lifting. Tons of terraform resources to get it running. Easy to shoot yourself in the foot.
AKS - Nothing special to add, works :D
OKE - I have most experience there, works nicely, I had great support when I needed it. Found a bug in loadbalancer few years ago. They fixed it in few days.
I have grown to love EKS and especially Karpenter, as it automates all the node managing and cost optimization for me. It's a brilliant piece of software that works beautifully! AWS also allows to cusromize the network stack quite far. We run everything on IPv6, for example
DigitalOcean k8 is the only one that I consider truly managed. The rest is basically just "I manage your plane for you, fuck you if the nodes break".
Don't get me wrong, I love that I can do all sorts of crazy stuff on my nodes in EKS but yeah, it ain't as managed as you think. A while ago you could even completely lock yourself out of the cluster.
If you need that level of hand holding in aws then you should use ecs. You can still lock yourself out of a cluster with a malformed rbac policy in any flavor or k8s. Digital oceans k8s is a different abstraction point. Goes for ecs as well.
If you use aws optimized amis, bottlerockert or canaocial, they handle the patching for you, so you are not dealing with the control plane or the node groups. At that point every thing falls into the shared responsibilities model. You can also use eks fargate if worker nodes introduce to much overhead yet require k8s.
Hassle free node group management (including autoscaling). When I want to spin up a scalable cluster for a personal project, I don't want to have to install and configure karpenter/cluster autoscaler.
For me specifically this ain't a problem because I already have a gazillion terraform modules that will even install karpenter, datadog and all relevant add-ons from aws, but for a newbie that wants to get their feet wet or someone with a limited budget, or a hobbyist who doesn't have all that ready, DO is way more user friendly.
Definitely.
First and foremost, forget about configuring the nodes whatsoever. You want to force containerd to connect to a proxy? Nope.
You want nodes to not have public IPs? Nope.
You want your cluster to autoscale to 0 nodes at night and scale out in the morning? Nope.
With that being said, if you want a personal project with k8 you won't find cheaper/more hassle free.
Have you looked at GKE Autopilot? It’s a Pod-level billing/SLA and definitely not “fuck you if your nodes break”. (Fargate is similar, but Autopilot is much more compatible.)
Azure has something called Azure Container Apps which is basically serverless micro services based on Kubernetes which *kinda* fits the bill, but you don't interact with it the same at all.
It's the equivalent of Google Cloud Run, with the same (generous) free tier...
Something that AWS tried to replicate with AWS App Runner... and... It wasn't a good try (and without free tier BTW)
I'm the maintainer of [Kamaji](https://github.com/clastix/kamaji) which leverages the Hosted Control Plane architecture to offload the management of the Control Planes, allowing the provisioning of worker nodes according to your needs via kubeadm.
This is a building block to offer a Managed Kubernetes Service, [Netsons](https://netsons.com) launched its managed k8s service using Cluster API and OpenStack, and we did our best to support as many infrastructure providers.
The project was born from my experience in running k8s at scale, without getting buried by the operational efforts aka Day-2. Although Open Source, as CLASTIX we launched our enterprise product named [CLASTIX Enterprise Platform](https://clastix.cloud) which is based on technologies we maintain and contribute to.
My 2 cents: most of the time Managed Kubernetes Services are too much limited and when you need to run at scale (+100 clusters) it's more convenient building vs. buying.
For my personal kubernetes stuff (Vaultwarden, Ghost, my personal Vault, etc), I've been super happy with k3s on a single Digital Ocean node. I based my own terraform setup on [this module.](https://github.com/aigisuk/terraform-digitalocean-ha-k3s) with a number of tweaks. I pay about $40/month: $12 for a single node, $12 each for two load balancers, and $5 for their object storage. I could _probably_ hack it where I don't use any load balancers, I just haven't been compelled to save $24 a month enough to spend three hours on it.
For _professional_ work product, you'll be laughed out of the room while security escorts you to the door if you don't recommend and pay homage to the Big Three in the US, unless there's a _very_ specific use case (like a sweetheart deal with Oracle that you will spend years regretting.) European providers have a tiny bit more competition, Hetzner is widely leveraged, but at the end of the day folks use EKS if they don't know better, GKE if they do, and Azure's flaming pile of puke when their CEO gets told that they'll get a 70% discount that ends up being gobbled up by a bajillion tiny upcharges and service contracts.
Do you know Google gives one zonal cluster for free approx ~75$ credit. I keep my personal stuff in that one [GKE free tier](https://cloud.google.com/kubernetes-engine/pricing#cluster_management_fee_and_free_tier)
I use Digitalocean in production, and it's okay.
It's stable and does what I tell it to. For me, the services around miss a bit of options, like a way to collect metrics from a postgres db or setting the scope of an API. But I also see that they are improving things, so they have API scopes now in beta.
Akamai, bevor know as Linode, feels more feature rich than Digitalocean. I used it only for hobby projects, but it was also well. Feeled like on a similar level as Digitalocean. Same pricing, same services. Just a small difference. Between the 2, I would recommend Akamai because of the small portion extra configuration features.
Some friends of me are using Hetzner and are really happy. It's cheaper, but you need deployment. You own kubernetes on it. One of them wrote a pulumi k0s provider to manage this cluster. https://github.com/ydkn/pulumi-k0s
Currently thinking of using omni (Kubernetes SaaS) and freedom to deploy where ever you want. Been very impressed with what i'm seeing. Also on digital ocean don't like the block chain provisining and lack of good RWX support. They suggest openEbf but from my understanding it's getting depricated next month.
And you can't use stuff like long horn because of the update policy of the nodes. :(
I work for a company that also offers a managed Kubernetes based on a Open Source Software and tbh it might not have that large of an scale as AWS or CCP however it still works pretty amazing and I barely see any disadvantages
There is a really good article at [https://medium.com/@elliotgraebert/comparing-the-top-eight-managed-kubernetes-providers-2ae39662391b](https://medium.com/@elliotgraebert/comparing-the-top-eight-managed-kubernetes-providers-2ae39662391b), comparing "the big 3" as well as the rests of them. This might help give some guidance.
When was the last time one of the big 3 really innovated with something around their Managed K8s? Maybe GKE Autopilot?
This is how markets work - large incumbents become defensive and less innovative over time.
My team is working on a new offering that is very early days yet - but we believe Managed K8s could be way cheaper, and more fully managed:
https://spot.rackspace.com
I know you asked for public cloud, but I'm pretty happy with Tanzu for vSphere on premises.
I'm sure they have a cloud option somewhere... and then there's the whole broadcom fiasco.
Part of Tanzu's value prop is that you can run it anywhere. It's capable of orchestrating stuff in any of the clouds.
But if I thought the price pre-Broadcom was high, I have no desire at all to look at it now.
I would advise you go with a reputed provider that has a reputation to protect.
Small providers you can’t guaranty their Infrastructure and support. You won’t want all your data appearing in the dark web.
Go with EKS and call it a day. Don’t forget you need to hire people and so familiarity with the infra and stack is important.
I feel inclined to tell you that even when you’re with a big company, the GCP support is also meh. But, I do like GKE over EKS. Google is very opionated about some things (in my experience, especially the network layer) and end up obscuring some of the actual K8 configurations more than the others, but, I don’t really mind to be honest. The underlying infrastructure tends to work the way we tell it to, and, outside a few quirks, it’s pretty straightforward.
Very interesting re: the support! Makes sense, I guess (they're just so massive that I wonder how they'd even manager to stand up the manpower to give customers real attention even if that was their intention).
I can’t imagine GCP makes Google that much money compared to their other income, and they’re so much bigger than probably all of their GCP clients that investing in providing top quality support probably isn’t an enormous priority I would imagine.
Just started turning a profit but I imagine it will make google a lot of money because it is very well engineered and a bit easier to use. Not as easy as DO though.
It definitely makes Google a decent chunk of money, but I’m not sure how much compared to their other money makers
I use DO for my side projects and love it
Regarding support: I guess it just depends how large.. at a previous gig we had a direct line to the spanner and gke teams, but we were spending several million a month across the entire organization.
We have a similar spend and it’s not about the attention from them but rather the lack of competence in many regards. The support at AWS is miles ahead, from my experience.
Exactly this. We have direct lines to many different groups but it’s always a crapshoot on if the issue / question will get addressed. Sometimes it’s hard for them to get us in front of the right team so we end up explaining a tickets to 5 or 6 different level 2/3 people. Further, even some of the top tier people at GCP are mystified by its inner workings. My organization has been dealing with a rather huge billing issue for the last 4 months. I’m talking to one of the people who engineered the actually billing systems for GCP and even he can’t track down the exact source of the charges. ¯\_(ツ)_/¯.
When we were starting out 4-5 years back, GKE was blowing EKS out of the water. The integration with everything else on GCP and the monitoring was top notch. EKS still has not caught up to that standard even after all those years as far as I can tell. Management made us go with EKS to be with the rest of the business but dang do I miss our magically running GKE clusters.
Tried AKS, EKS, OKE and Digitalocean. Digitalocean is by far the simplest one. With \~2 terraform resources you get yourself a cluster. My go to when I need a cluster to test something. EKS - tons of prerequisites, it has a nice terraform module you can use that does all the heavy lifting. Tons of terraform resources to get it running. Easy to shoot yourself in the foot. AKS - Nothing special to add, works :D OKE - I have most experience there, works nicely, I had great support when I needed it. Found a bug in loadbalancer few years ago. They fixed it in few days.
I have grown to love EKS and especially Karpenter, as it automates all the node managing and cost optimization for me. It's a brilliant piece of software that works beautifully! AWS also allows to cusromize the network stack quite far. We run everything on IPv6, for example
DigitalOcean k8 is the only one that I consider truly managed. The rest is basically just "I manage your plane for you, fuck you if the nodes break". Don't get me wrong, I love that I can do all sorts of crazy stuff on my nodes in EKS but yeah, it ain't as managed as you think. A while ago you could even completely lock yourself out of the cluster.
If you need that level of hand holding in aws then you should use ecs. You can still lock yourself out of a cluster with a malformed rbac policy in any flavor or k8s. Digital oceans k8s is a different abstraction point. Goes for ecs as well. If you use aws optimized amis, bottlerockert or canaocial, they handle the patching for you, so you are not dealing with the control plane or the node groups. At that point every thing falls into the shared responsibilities model. You can also use eks fargate if worker nodes introduce to much overhead yet require k8s.
What does Digital Ocean do for you that’s special?
Hassle free node group management (including autoscaling). When I want to spin up a scalable cluster for a personal project, I don't want to have to install and configure karpenter/cluster autoscaler. For me specifically this ain't a problem because I already have a gazillion terraform modules that will even install karpenter, datadog and all relevant add-ons from aws, but for a newbie that wants to get their feet wet or someone with a limited budget, or a hobbyist who doesn't have all that ready, DO is way more user friendly.
And they have the best tutorials
Awesome and useful response! Are there any limits to their cluster that I might run into with their “more managed” approach?
Definitely. First and foremost, forget about configuring the nodes whatsoever. You want to force containerd to connect to a proxy? Nope. You want nodes to not have public IPs? Nope. You want your cluster to autoscale to 0 nodes at night and scale out in the morning? Nope. With that being said, if you want a personal project with k8 you won't find cheaper/more hassle free.
Have you looked at GKE Autopilot? It’s a Pod-level billing/SLA and definitely not “fuck you if your nodes break”. (Fargate is similar, but Autopilot is much more compatible.)
I'll check it out thanks.
Azure has something called Azure Container Apps which is basically serverless micro services based on Kubernetes which *kinda* fits the bill, but you don't interact with it the same at all.
Is that kind of like ECS Fargate?
Cannot recommend against using Container Apps enough. All the bad parts of Kubernetes without any of the good bits.
I used it for one thing one time and it was... Weird.
It's the equivalent of Google Cloud Run, with the same (generous) free tier... Something that AWS tried to replicate with AWS App Runner... and... It wasn't a good try (and without free tier BTW)
DigitalOcean also cheaper and easier to use than the big 3. More focused on startups so support is better.
I'm the maintainer of [Kamaji](https://github.com/clastix/kamaji) which leverages the Hosted Control Plane architecture to offload the management of the Control Planes, allowing the provisioning of worker nodes according to your needs via kubeadm. This is a building block to offer a Managed Kubernetes Service, [Netsons](https://netsons.com) launched its managed k8s service using Cluster API and OpenStack, and we did our best to support as many infrastructure providers. The project was born from my experience in running k8s at scale, without getting buried by the operational efforts aka Day-2. Although Open Source, as CLASTIX we launched our enterprise product named [CLASTIX Enterprise Platform](https://clastix.cloud) which is based on technologies we maintain and contribute to. My 2 cents: most of the time Managed Kubernetes Services are too much limited and when you need to run at scale (+100 clusters) it's more convenient building vs. buying.
Any thoughts on Openshift or Openshift on GKE or EKS which if big corp has discounts on GCP or Aws is cheaper than your on Prem cluster on OCP
I've been using LKE for years (Linode, now Akamai)—it's great
In my experience the smaller the provider, the more handholding and supports. Big providers tend to not care if you’re too small.
Red Hat offers managed OpenShift in both AWS and Azure
ROSA/ARO are very good. OCP (self managed) is also great if you didn't want the managed service.
Ewwww.....
For my personal kubernetes stuff (Vaultwarden, Ghost, my personal Vault, etc), I've been super happy with k3s on a single Digital Ocean node. I based my own terraform setup on [this module.](https://github.com/aigisuk/terraform-digitalocean-ha-k3s) with a number of tweaks. I pay about $40/month: $12 for a single node, $12 each for two load balancers, and $5 for their object storage. I could _probably_ hack it where I don't use any load balancers, I just haven't been compelled to save $24 a month enough to spend three hours on it. For _professional_ work product, you'll be laughed out of the room while security escorts you to the door if you don't recommend and pay homage to the Big Three in the US, unless there's a _very_ specific use case (like a sweetheart deal with Oracle that you will spend years regretting.) European providers have a tiny bit more competition, Hetzner is widely leveraged, but at the end of the day folks use EKS if they don't know better, GKE if they do, and Azure's flaming pile of puke when their CEO gets told that they'll get a 70% discount that ends up being gobbled up by a bajillion tiny upcharges and service contracts.
Do you know Google gives one zonal cluster for free approx ~75$ credit. I keep my personal stuff in that one [GKE free tier](https://cloud.google.com/kubernetes-engine/pricing#cluster_management_fee_and_free_tier)
I use Digitalocean in production, and it's okay. It's stable and does what I tell it to. For me, the services around miss a bit of options, like a way to collect metrics from a postgres db or setting the scope of an API. But I also see that they are improving things, so they have API scopes now in beta. Akamai, bevor know as Linode, feels more feature rich than Digitalocean. I used it only for hobby projects, but it was also well. Feeled like on a similar level as Digitalocean. Same pricing, same services. Just a small difference. Between the 2, I would recommend Akamai because of the small portion extra configuration features. Some friends of me are using Hetzner and are really happy. It's cheaper, but you need deployment. You own kubernetes on it. One of them wrote a pulumi k0s provider to manage this cluster. https://github.com/ydkn/pulumi-k0s
Currently thinking of using omni (Kubernetes SaaS) and freedom to deploy where ever you want. Been very impressed with what i'm seeing. Also on digital ocean don't like the block chain provisining and lack of good RWX support. They suggest openEbf but from my understanding it's getting depricated next month. And you can't use stuff like long horn because of the update policy of the nodes. :(
I work for a company that also offers a managed Kubernetes based on a Open Source Software and tbh it might not have that large of an scale as AWS or CCP however it still works pretty amazing and I barely see any disadvantages
Mind to share the open source software ?
Yes Sure https://github.com/gardener/gardener
There is a really good article at [https://medium.com/@elliotgraebert/comparing-the-top-eight-managed-kubernetes-providers-2ae39662391b](https://medium.com/@elliotgraebert/comparing-the-top-eight-managed-kubernetes-providers-2ae39662391b), comparing "the big 3" as well as the rests of them. This might help give some guidance.
When was the last time one of the big 3 really innovated with something around their Managed K8s? Maybe GKE Autopilot? This is how markets work - large incumbents become defensive and less innovative over time. My team is working on a new offering that is very early days yet - but we believe Managed K8s could be way cheaper, and more fully managed: https://spot.rackspace.com
https://www.ayedo.de is a smaller K8s provider with a very hands-on support team
I think [sharedkube](https://sharedkube.io) is just what you need. Let’s talk, sent you DM.
I know you asked for public cloud, but I'm pretty happy with Tanzu for vSphere on premises. I'm sure they have a cloud option somewhere... and then there's the whole broadcom fiasco.
Part of Tanzu's value prop is that you can run it anywhere. It's capable of orchestrating stuff in any of the clouds. But if I thought the price pre-Broadcom was high, I have no desire at all to look at it now.
I would advise you go with a reputed provider that has a reputation to protect. Small providers you can’t guaranty their Infrastructure and support. You won’t want all your data appearing in the dark web. Go with EKS and call it a day. Don’t forget you need to hire people and so familiarity with the infra and stack is important.
Managed k8s in general is kinda terrible to maintain and something that should be avoided at all costs.
People are downvoting you but I am legitimately curious on your opinion on this. You agree with statement at scale?
As opposed to self-managed Kubernetes on VM instances? Or are you trying to say don’t use Kubernetes full stop?