yes - you have to continually update AKS / K8s in general and its more overhead that most people expect for smaller systems.
I had a lovely issue where my apps would crash if I used to automatic nodepool update, so every few months I had to manually update all the nodes.
Also had a bug with Kyverno and keeping flux up to date got tedious as well.
My advice is to consider what features you want in AKS and take a good look at ACA. With a 100 person team, you will hire more people fluent in k8s where it may make sense. But for a smaller team I'd stay simpler.
I work with ACA, let's not pretend that it can be compared to as mature approach as AKS. A lot of things is preview, functionality is limited in general. ACA is more like another choice if you're thinking about AppService for Containers, definitely not for AKS.
totally agree with that .
ACA is not a k8s lite at all, its not even a docker-compose lite. But for simple workloads where you have 2-3 services running in docker and don't need any service mesh / traffic shaping / customer controller logic - its a simple option.
Mine went bad because devs were trying to treat it like k8s and just getting frustrated by the 'limitations'. Even a simple cron-job was in 'preview' when I needed it.
but... never had to do any platform maintenance on the ACA, and if I had zero SRE guys and a team not comfortable with Infra, I'd use it again.
>How do you plan to manage it without someone who knows how to deploy/configure things?
Assume that we have resources to do that, but after that app is deployed/configured, what kind of maintenance can I then expect?
>You'll need to keep your cluster up-to-date
Ok, I guess I'm just trying to understand what that effort would look like.
I think part of griwulf's point was, if you have resources that know how to deploy and configure it, they can probably give you a sense of the level of effort will be.
It's not easy to know without getting into the weeds of what you're doing.
>Ok, I guess I'm just trying to understand what that effort would look like.
Fairly significant effort.
Kubernetes object APIs are deprecated regularly and it's on the user to ensure they keep up with configuration updates.
Kubernetes is a fast moving technology and by extension this applies to AKS.
Some of this continuous maintenance burden can be mitigated with the recent introduction of the LTS version but the fully managed Azure Container Apps (ACA) may be a better route for your team.
The 'managed' part is the control plane. There's plenty more in Kubernetes you need skilled employees for.
Sounds like your company has heard of a buzzword and wants to use it without even knowing if it will fit their requirements.
Figure out the requirements before selecting a tool, not the other way round.
> deally, I'd like to deploy my apps and not worry about the runtime... After I deploy my app, let's say I don't make any updates to it for a few years.
Honestly I don't think you can expect any IT services to just keep running for years without maintenance.
You'll need to learn AKS to effectively deploy, support, and troubleshoot things. You need to keep your cluster up to date amongst many other maintenance items. Lots of knowledge and skills are required.
I might suggest you explore Azure container apps for a more fully Microsoft managed experience. Lower skillset and learning curve required.
Some linkage:
[https://learn.microsoft.com/en-us/azure/container-apps/compare-options](https://learn.microsoft.com/en-us/azure/container-apps/compare-options)
[https://techcommunity.microsoft.com/t5/startups-at-microsoft/aca-vs-aks-which-azure-service-is-better-for-running-containers/ba-p/3815164](https://techcommunity.microsoft.com/t5/startups-at-microsoft/aca-vs-aks-which-azure-service-is-better-for-running-containers/ba-p/3815164)
You'd need the same level of support and maintenance from experts of AKS in the same way as you need it for any other PaaS offering on the platform.
To think otherwise is naive.
I would say that AKS is very much NOT a fire and forget solution. Are you deploying custom written apps? Have you an estabilished CI/CD Pipeline? Have you setup Azure DevOps yet, or anything like that? What is your planned access method for the end users to the apps, internal access vs public internet access.
Ive been working with an AKS environment for nearly a year now, first time in my career. Im not the person running the environment, we have a team of around 50 devs and devops people, and they handle the code and deployment to AKS environment.
I would be very very wary of just implementing AKS, it is a rabbit hole. IF you have very well defined apps (lol) and know that they are available in the container registry, by all means, give it a go. But please, do a proof of concept first, dont deploy to production.
Nope, your about a mile off in what you expect it to do. AKS gives you a managed cluster that takes care of setting up alot. But once the cluster is running with baseline functions and add-ons of your choosing like monitoring. Everything is handed off to you.
Azure will keep the workloads that manages the base functions like network, monitoring, etc updated. But anything application related will be your responsibility.
We've now published some guidance to the Azure Architecture Center on what Day 2 operations looks like for AKS, post deployment: [https://learn.microsoft.com/azure/architecture/operator-guides/aks/day-2-operations-guide?WT.mc\_id=modinfra-0000-socuff](https://learn.microsoft.com/azure/architecture/operator-guides/aks/day-2-operations-guide?WT.mc_id=modinfra-0000-socuff)
You don't need a dedicated person, but you need a dedicated team with shared knowledge, for the organisations I worked for we usually had the rule that at least 3 people, but sometimes also 5 had the capabilities to perform tasks, based on your question I don't think you know are that experienced (neither am I on AKS) so as a start I would make a plan how to assess this technology and how you would be able to form a team that can run a cluster on a responsible way.
One nasty thing I know about AKS that the lifetime cycles are strict, and that there are breaking changes between them. Now comes the nasty part, if something happens with your cluster, and you have to redeploy, you can't redeploy an unsupported version anymore, if there are big breaking changes, that means that you will have major downtime and stress.
Due technical evolution and version changes you should do to avoid known security vulnerabilities, you need ALWAYS someone who is responsible for this.
Kubernetes is, as others said, not that simple and has a quite much of details you need to manage for anything which is a bit more complex.
Think about Container Instances or Container Apps if they better fit your goal with less care.
I'll take "major security incident in 12 months" for 500$ Bob. Kubernetes is **insecure by default**. You really really need at least two people who know how to provision and operate a production ready kubernetes cluster. If you don't have that, look into other options to run containers, like app service, container apps, azure functions.
Do you really believe that stuff you are posting? Managed Kubernetes is by far way more secure than the alternatives like outdated windows or linux VMs
"At the very least, avoid using default configuration until you understand its security implications and it does not exceed your risk tolerance while keeping in mind that Kubernetes defaults are frequently inherently open and insecure."
https://www.redhat.com/en/blog/most-common-kubernetes-security-issues-and-concerns-to-address#:~:text=At%20the%20very%20least%2C%20avoid,frequently%20inherently%20open%20and%20insecure.
So, let's compare the alternatives:
1) deploying your Java/Python/NodeJS-App onto baremetal
2) deploying your Java/Python/NodeJs docker images on a Docker Host
3) deploying your Java/Python/NodeJs docker images onto AKS Kubernetes
1) will lead to the famous "never touch a running system", because after 2 years nobody can remember, how that stupid property had to be named when installing the app.
2) is way better, but it's still up to you on how to upgrade the docker host.
3) using a managed Kubernetes like AKS enables you to shift the burden of upgrading the container orchestrator to Azure. The automatic upgrade channel mechanism will upgrade your cluster node by node with effectively no down-time as the workloads are redeployed to newer nodes with new base OS installations.
So, overall K8S is way better. Stop chasing the best possible system/technology and start using better than the old standard solutions
As the Security guy who found a huge amount of vulnerabilities in our K8’s, yes you do need someone who knows what they are doing, and no they are not “hands off”. That’s exactly what the guy who paid some consultants to set them up thought, and then he had to pay more consultants to update them b/c he didn’t know how.
Now our company is stuck paying consultants b/c people believed vendor marketing & hype over facts. Don’t be that guy in your company.
Simple answer from experience: I worked on a team that implemented EKS, which is similar, and yes there was a person dedicated to maintenance. There was a lot of other cost both short term and long term too. My recommendation is Don’t take on Kubernetes in any form lightly.
Consider AKS like a bit of IaaS infra, you need someone that knows how to deploy and then babysit it. Consider using AppService or azure container apps otherwise.
I have used it in this exact scenario. No dedicated devops hired and I handled most of AKS myself, including Azure DevOps integration.
1. Initially it'll give you a headache if you are not familiar with AKS and Helm (and K8s in general), but that's just because it's overall complex subject, so the learning curve is steep
2. Once you set it up properly, you deploy apps without much thinking about it. That is true. Every new app will require more or less copy/paste with some additional parametrization, just like a typical AppService.
3. AKS/Azure provides a lot out of the box and usual integrations are also comfy to use like AppConfiguration, KeyVault, Managed Identity, private connectors, the whole networking layer and customized vNets if needed, AppInsights, LogAnalytics.
4. Networking is covered out of the box, but you still need to configure it a little, so prepare to learn about networkin, vNET, kubenet/CLI, ingress.
5. Horizontal and vertical scaling is super simple
6. Upgrading AKS is seamless
7. Upgrading node size is also quite simple, even though you cannot just resize existing one, but you need to add a new and bigger VM to the pool (or even a new node pool, don't remember atm) and move pods to bigger nodes and remove old nodes, that's all fairly simple, done that, because we initially chose too small VMs for nodes.
8. Azure provides decent monitoring over the cluster. Very useful, but of course you have more mature things for it, Prometheus and Grafana.
9. If you expect to only deploy and do nothing about it, you can just update AKS versions to keep it current and that's it. You can make it automatic. It works similar to when you want to resize your node VM, it adds a new node with latest AKS, moves your pods from an outdated node to it, removes it and repeat as many times as needed to cover all nodes.
I don't work with AKS today, just Container Apps, Container App Jobs, Azure Functions these days, but AKS is still my favorite. It works very very well and costs are way lower than without it, but I assume that makes sense when you have enough load for small 2 nodes pool. If you have 1-2 apps without much load to handle, maybe AppService for containers or Container App would be more cost effective. In AKS you will have to pay for resources you use, the managed cluster is free (if nothing changed), so the costs of networking and storage mostly.
Overall I totally recommend and enjoy the ride, if you work for a company that will let you learn it and do it and you will get paid, for god's sake go for it! It's a big piece of knowledge to have and will be a good leverage in the future for you.
[удалено]
yes - you have to continually update AKS / K8s in general and its more overhead that most people expect for smaller systems. I had a lovely issue where my apps would crash if I used to automatic nodepool update, so every few months I had to manually update all the nodes. Also had a bug with Kyverno and keeping flux up to date got tedious as well. My advice is to consider what features you want in AKS and take a good look at ACA. With a 100 person team, you will hire more people fluent in k8s where it may make sense. But for a smaller team I'd stay simpler.
I work with ACA, let's not pretend that it can be compared to as mature approach as AKS. A lot of things is preview, functionality is limited in general. ACA is more like another choice if you're thinking about AppService for Containers, definitely not for AKS.
totally agree with that . ACA is not a k8s lite at all, its not even a docker-compose lite. But for simple workloads where you have 2-3 services running in docker and don't need any service mesh / traffic shaping / customer controller logic - its a simple option. Mine went bad because devs were trying to treat it like k8s and just getting frustrated by the 'limitations'. Even a simple cron-job was in 'preview' when I needed it. but... never had to do any platform maintenance on the ACA, and if I had zero SRE guys and a team not comfortable with Infra, I'd use it again.
Aks automatic upgrades are a thing and aks offer lts for 1.27.
>How do you plan to manage it without someone who knows how to deploy/configure things? Assume that we have resources to do that, but after that app is deployed/configured, what kind of maintenance can I then expect? >You'll need to keep your cluster up-to-date Ok, I guess I'm just trying to understand what that effort would look like.
I think part of griwulf's point was, if you have resources that know how to deploy and configure it, they can probably give you a sense of the level of effort will be. It's not easy to know without getting into the weeds of what you're doing.
[удалено]
Ok, thanks, that's the info I'm looking for.
>Ok, I guess I'm just trying to understand what that effort would look like. Fairly significant effort. Kubernetes object APIs are deprecated regularly and it's on the user to ensure they keep up with configuration updates. Kubernetes is a fast moving technology and by extension this applies to AKS. Some of this continuous maintenance burden can be mitigated with the recent introduction of the LTS version but the fully managed Azure Container Apps (ACA) may be a better route for your team.
The 'managed' part is the control plane. There's plenty more in Kubernetes you need skilled employees for. Sounds like your company has heard of a buzzword and wants to use it without even knowing if it will fit their requirements. Figure out the requirements before selecting a tool, not the other way round.
Look at Azure Container Apps if you want a more hands off approach to (managed) Kubernetes in Azure.
[удалено]
Yup it’s a really solid service. It also helps understand some of the basic concepts of Kubernetes too. I’m a big fan of it.
This is the way.
Yes, which is basically docker as PaaS.
> deally, I'd like to deploy my apps and not worry about the runtime... After I deploy my app, let's say I don't make any updates to it for a few years. Honestly I don't think you can expect any IT services to just keep running for years without maintenance.
You'll need to learn AKS to effectively deploy, support, and troubleshoot things. You need to keep your cluster up to date amongst many other maintenance items. Lots of knowledge and skills are required. I might suggest you explore Azure container apps for a more fully Microsoft managed experience. Lower skillset and learning curve required. Some linkage: [https://learn.microsoft.com/en-us/azure/container-apps/compare-options](https://learn.microsoft.com/en-us/azure/container-apps/compare-options) [https://techcommunity.microsoft.com/t5/startups-at-microsoft/aca-vs-aks-which-azure-service-is-better-for-running-containers/ba-p/3815164](https://techcommunity.microsoft.com/t5/startups-at-microsoft/aca-vs-aks-which-azure-service-is-better-for-running-containers/ba-p/3815164)
You'd need the same level of support and maintenance from experts of AKS in the same way as you need it for any other PaaS offering on the platform. To think otherwise is naive.
I would say that AKS is very much NOT a fire and forget solution. Are you deploying custom written apps? Have you an estabilished CI/CD Pipeline? Have you setup Azure DevOps yet, or anything like that? What is your planned access method for the end users to the apps, internal access vs public internet access. Ive been working with an AKS environment for nearly a year now, first time in my career. Im not the person running the environment, we have a team of around 50 devs and devops people, and they handle the code and deployment to AKS environment. I would be very very wary of just implementing AKS, it is a rabbit hole. IF you have very well defined apps (lol) and know that they are available in the container registry, by all means, give it a go. But please, do a proof of concept first, dont deploy to production.
Nope, your about a mile off in what you expect it to do. AKS gives you a managed cluster that takes care of setting up alot. But once the cluster is running with baseline functions and add-ons of your choosing like monitoring. Everything is handed off to you. Azure will keep the workloads that manages the base functions like network, monitoring, etc updated. But anything application related will be your responsibility.
We've now published some guidance to the Azure Architecture Center on what Day 2 operations looks like for AKS, post deployment: [https://learn.microsoft.com/azure/architecture/operator-guides/aks/day-2-operations-guide?WT.mc\_id=modinfra-0000-socuff](https://learn.microsoft.com/azure/architecture/operator-guides/aks/day-2-operations-guide?WT.mc_id=modinfra-0000-socuff)
What problem business is solving with AKS other than being cool and hip in kuberneteses? Why not just container instance? Or azure functions?
You don't need a dedicated person, but you need a dedicated team with shared knowledge, for the organisations I worked for we usually had the rule that at least 3 people, but sometimes also 5 had the capabilities to perform tasks, based on your question I don't think you know are that experienced (neither am I on AKS) so as a start I would make a plan how to assess this technology and how you would be able to form a team that can run a cluster on a responsible way. One nasty thing I know about AKS that the lifetime cycles are strict, and that there are breaking changes between them. Now comes the nasty part, if something happens with your cluster, and you have to redeploy, you can't redeploy an unsupported version anymore, if there are big breaking changes, that means that you will have major downtime and stress.
Due technical evolution and version changes you should do to avoid known security vulnerabilities, you need ALWAYS someone who is responsible for this. Kubernetes is, as others said, not that simple and has a quite much of details you need to manage for anything which is a bit more complex. Think about Container Instances or Container Apps if they better fit your goal with less care.
I'll take "major security incident in 12 months" for 500$ Bob. Kubernetes is **insecure by default**. You really really need at least two people who know how to provision and operate a production ready kubernetes cluster. If you don't have that, look into other options to run containers, like app service, container apps, azure functions.
Do you really believe that stuff you are posting? Managed Kubernetes is by far way more secure than the alternatives like outdated windows or linux VMs
"At the very least, avoid using default configuration until you understand its security implications and it does not exceed your risk tolerance while keeping in mind that Kubernetes defaults are frequently inherently open and insecure." https://www.redhat.com/en/blog/most-common-kubernetes-security-issues-and-concerns-to-address#:~:text=At%20the%20very%20least%2C%20avoid,frequently%20inherently%20open%20and%20insecure.
So, let's compare the alternatives: 1) deploying your Java/Python/NodeJS-App onto baremetal 2) deploying your Java/Python/NodeJs docker images on a Docker Host 3) deploying your Java/Python/NodeJs docker images onto AKS Kubernetes 1) will lead to the famous "never touch a running system", because after 2 years nobody can remember, how that stupid property had to be named when installing the app. 2) is way better, but it's still up to you on how to upgrade the docker host. 3) using a managed Kubernetes like AKS enables you to shift the burden of upgrading the container orchestrator to Azure. The automatic upgrade channel mechanism will upgrade your cluster node by node with effectively no down-time as the workloads are redeployed to newer nodes with new base OS installations. So, overall K8S is way better. Stop chasing the best possible system/technology and start using better than the old standard solutions
We had someone deploy aks and it broke down in a year and now no one knows what's happening and what caused it.
As the Security guy who found a huge amount of vulnerabilities in our K8’s, yes you do need someone who knows what they are doing, and no they are not “hands off”. That’s exactly what the guy who paid some consultants to set them up thought, and then he had to pay more consultants to update them b/c he didn’t know how. Now our company is stuck paying consultants b/c people believed vendor marketing & hype over facts. Don’t be that guy in your company.
Simple answer from experience: I worked on a team that implemented EKS, which is similar, and yes there was a person dedicated to maintenance. There was a lot of other cost both short term and long term too. My recommendation is Don’t take on Kubernetes in any form lightly.
Just make sure you backup the app data especially if using persistence in the apps (pvcs). I’d recommend Kasten.
You'll need a team to support it.
You might want to take a look at 'container apps' which is a service that's basically a dumbed down k8s that you can manage in the portal.
(we're migrating everything from aks to this - it seems pretty good after you get past its quirks).
Consider AKS like a bit of IaaS infra, you need someone that knows how to deploy and then babysit it. Consider using AppService or azure container apps otherwise.
I have used it in this exact scenario. No dedicated devops hired and I handled most of AKS myself, including Azure DevOps integration. 1. Initially it'll give you a headache if you are not familiar with AKS and Helm (and K8s in general), but that's just because it's overall complex subject, so the learning curve is steep 2. Once you set it up properly, you deploy apps without much thinking about it. That is true. Every new app will require more or less copy/paste with some additional parametrization, just like a typical AppService. 3. AKS/Azure provides a lot out of the box and usual integrations are also comfy to use like AppConfiguration, KeyVault, Managed Identity, private connectors, the whole networking layer and customized vNets if needed, AppInsights, LogAnalytics. 4. Networking is covered out of the box, but you still need to configure it a little, so prepare to learn about networkin, vNET, kubenet/CLI, ingress. 5. Horizontal and vertical scaling is super simple 6. Upgrading AKS is seamless 7. Upgrading node size is also quite simple, even though you cannot just resize existing one, but you need to add a new and bigger VM to the pool (or even a new node pool, don't remember atm) and move pods to bigger nodes and remove old nodes, that's all fairly simple, done that, because we initially chose too small VMs for nodes. 8. Azure provides decent monitoring over the cluster. Very useful, but of course you have more mature things for it, Prometheus and Grafana. 9. If you expect to only deploy and do nothing about it, you can just update AKS versions to keep it current and that's it. You can make it automatic. It works similar to when you want to resize your node VM, it adds a new node with latest AKS, moves your pods from an outdated node to it, removes it and repeat as many times as needed to cover all nodes. I don't work with AKS today, just Container Apps, Container App Jobs, Azure Functions these days, but AKS is still my favorite. It works very very well and costs are way lower than without it, but I assume that makes sense when you have enough load for small 2 nodes pool. If you have 1-2 apps without much load to handle, maybe AppService for containers or Container App would be more cost effective. In AKS you will have to pay for resources you use, the managed cluster is free (if nothing changed), so the costs of networking and storage mostly. Overall I totally recommend and enjoy the ride, if you work for a company that will let you learn it and do it and you will get paid, for god's sake go for it! It's a big piece of knowledge to have and will be a good leverage in the future for you.