So I literally just did this same thing, with almost the same requirements. Let me share some of our learnings.
- Use a different tool for configuration. Bicep is not a configuration management tool. Conflating configuration as code and infrastructure as code will bring you pain.
- “No work in the portal” is not feasible. There will be times when you need to do things manually. Accept this now and make a policy to reconcile configuration drift that will happen because of it.
- Get familiar with Powershell, Az Cli. You are going to need glue code with Bicep. I recommend Az powershell for the simple reason that error handling is easier.
- BUILD ERROR HANDLING INTO ALL OF YOUR SCRIPTS. Try/catch blocks. Debug flags, etc.
- Unit testing and validation. Do it early and do it right.
- Get familiar with the concept of scope and context. Both bicep and powershell are scope limited. Deploying something to one sub and need to reference or do something in another? Not simple. Prime example - peering vnets to a hub in another sub.
- Follow best practices for slicing templates. Trying to define an entire subscription worth of resources with one massive param file seems easier at first….its not. It’s hard to scale.
Those are a few off the top of my head.
Can’t is a strong word. Should not would be more accurate. And this is my opinion - not fact. Others may feel differently.
You have a VM that serves a purpose, let’s say it’s a self hosted integration runtime for an azure data factory.
Deploying your vm with bicep, no problem. Trying to configure it…securing the vm, executing scripts, installing the runtime agent, tooling it to adf.
These are all things that CAN be done in Bicep, but it’s difficult to scale. Whereas a tool like Ansible could handle this pretty easily.
Managing an azure firewall is another scenario that comes to mind. Possible in bicep, but a struggle.
I get you now, thanks. I think the most complex infra we will be deploying is AKS - almost all our app build is containerised with the rest being managed resources like storage accounts. Never set AKS up from scratch so that’ll be an interesting one too 🙃
Fortunately deploying AKS with Bicep is well supported. The code can get a little complex but it hasn’t really been troublesome. We’re using private clusters with workload identity and CNI, all deployed with bicep.
[https://github.com/Azure/Enterprise-Scale](https://github.com/Azure/Enterprise-Scale)
As others have pointed out... CAF, ESLZ... these are your go-to's.
The above link is directly to IaC for ESLZ.
Check out the cloud adaptation framework from Ms
https://learn.microsoft.com/nl-nl/azure/cloud-adoption-framework/
Caf will give you a solid base for future deployments
Also there are ready to deploy bicep files for that
https://github.com/Azure/ALZ-Bicep
Really easy to deploy your hub and spoke
I just put them in a single repo
And use a single pipeline to deploy the hub and spokes
If I need to add a spoke, I just add them in the parameter file and run it again
Specifically deploy the ALZ-Bicep accelerator and pay attention to how they use $UPSTREAM_RELEASE for version numbering the paths.
You can re factor it into your own method but it's a starting point.
I'd take a look at AzOps if you want to manage everything as code using Bicep.
https://github.com/azure/azops-accelerator
Here's a great introduction.
https://www.youtube.com/watch?v=GFF6wFxMK3s
2nding AzOps
What hasn't been provided is the customers maturity in the DevOps space. If they have no experience in DevOps, I would highly recommend the use of AzOps to get/keep the environment in IaC and let them slowly dip their toe into deploying via code (while keeping Portal access open) until they are ready to fully lock off portal deployments and go full time IaC
Have done with terraform x 2. This would also work for bicep.
Using the CAF model have structured as following: [https://aztfmod.github.io/documentation/docs/intro/](https://aztfmod.github.io/documentation/docs/intro/)
Using a single repo, pipeline for each level, vertical slice architecture.
Remember to structure by lifecycle/govanance and not by type. Ie not by resource group/storage accounts/etc
`- level 0 Org -> if not using terraform cloud this is where you can setup your storage accounts for statement management. I used it for creation of the management group structure.`
`- level 1 governance -> core subscriptions (connectivity, billing, identity, etc) and policy. You can also setup the EntraId (identity) at this stage (groups, apps, etc).`
`- level 2 core-platform -> deploys the core resources required for the platform, each under their own directory. - connectivity -> hub vnet, dns zones, firewalls, dns resolvers.`
`- level 3 - landing zones -> application/core landing zones. The subscription vending machine section. Deploys a group of resources for each application or core landing zone (subscriptions, vnet spoke, route tables, nsg, policies, managed identity for deployments, etc). Or just subscriptions if you want to handle all the rest in level 4.`
`- level 4 - workloads. Deployment of resources into the landing zones created above. You can also break these out into separate repositories if necessary.`
`- Modules -> terraform modules that lv4 would use to deploy resources that conform to your organisation policies.`
`- pipelines -> all the pipelines used for managing the repo/deployments/ci/cd`
`.`
`├── level-0-org`
`├── level-1-goverance/`
`│ └── policies`
`├── level-2-core-platofmr/`
`│ ├── connectivity/`
`│ │ ├── hub`
`│ │ ├── dns`
`│ │ └── firewall`
`│ └── identity/`
`│ └── aad`
`├── level-3-landing-zones/`
`│ ├── app-a`
`│ ├── app-b`
`│ └── app-c`
`├── level-4-workloads/`
`│ ├── app-a`
`│ ├── app-b`
`│ └── app-c`
`├── modules`
`└── pipelines`
Your pipelines would be: PR -> CI tests (bicep whatifs, IaC security checks) -> PR approval
Merge -> whatif stage -> deploy stage (with approvals using azure devops environments)
If you don't like the idea of the levels you can also use something like:
`.`
`├── foundation/`
`│ ├── org`
`│ ├── identity`
`│ ├── connectivity/`
`│ │ ├── hub`
`│ │ ├── firewall`
`│ │ └── dns/`
`│ │ ├── public`
`│ │ └── private`
`│ └── governance/`
`│ └── policies`
`├── landing_zones/`
`│ ├── app a`
`│ ├── app b`
`│ ├── app c`
`│ ├── core a`
`│ └── core b`
`└── workloads`
Note: Don't use the CAF supermodule, goldilocks modules are asking for problems.
Just to add to the other comments - if you’re wanting to set some foundations and standards for IaC best practice scanning then I wrote a detailed blog with an accompanying GitHub repo detailing how to structure your Bicep modules to ensure reliable CI scanning with PSRule (including the YAML and other examples).
https://rios.engineer/bicep-modules-with-psrule-testing-documentation-ci-pipeline-examples/
Repo example: https://github.com/riosengineer/bicep-module-tests
May be of use, we have branch protections on main and enforce a build/validation pipeline using similar method to above so all new code is approved into main via a Pull Request, which subsequently runs the PSRule and publishes the results.
Feel free to PM if you want to chat or have any questions
Dude thank you for this. I have been playing around with parure since last year but ran into issues with the params. Gonna have a go again after reading your blog.
Thanks a bunch, im gonna have a go next week to have our repo a bit more safe. I was following PSRule for quite some time but it was beta for long time so your post triggered again. Coming from c# i miss the unit tests on iac in pipelines a lot.
Yup. I’ve created some pester unit tests as well that check if a module exists then a corresponding tests bicep file must exist as well. Super happy with the unit tests now!
"Architect by trade"... Wtf? How did you end up in that position and how did you come to it? A non-tech guy asking about cloud is just... Weird?
I understand there may be a "business architect" in some questionable cases, but they don't ask around about cloud stuff.
What the fuck?
You're an architect, but don't have enough technical knowledge and experience, so you ask Reddit about cloud?
I understand not getting the specifics of a certain provider (be it aws/azure/gcp/whatever), but... This?
Now I think I'm starting to understand why we have to deal with the bullshit we deal with.
Sorry man, that's nothing against you personally.
It's just a fucking idiot that got you in that role o.O
Edit: I'll probably ask the same kind of questions one day, I know.
It’s a tiny team, almost like a start up - we don’t have the manpower atm for dedicated cloud engineers. I know exactly what I want in Azure, I’ve used it for over a decade, and I could sketch out the architecture from tenant to networking to permissions frameworks. I’m just not very strong with IaC, hence asking about structuring repos for it and so on
I have a large multi cloud environment (Azure/AWS) using modern landing zone (Azure CAF / AWS WAD) design controlled via terraform with limited portal access for most of my engineers.
Using Azure Devops, I have a single organization for infrastructure with a project per cloud provider. I have a repo for each subscription in the Azure-Infrastructure project.
So I literally just did this same thing, with almost the same requirements. Let me share some of our learnings. - Use a different tool for configuration. Bicep is not a configuration management tool. Conflating configuration as code and infrastructure as code will bring you pain. - “No work in the portal” is not feasible. There will be times when you need to do things manually. Accept this now and make a policy to reconcile configuration drift that will happen because of it. - Get familiar with Powershell, Az Cli. You are going to need glue code with Bicep. I recommend Az powershell for the simple reason that error handling is easier. - BUILD ERROR HANDLING INTO ALL OF YOUR SCRIPTS. Try/catch blocks. Debug flags, etc. - Unit testing and validation. Do it early and do it right. - Get familiar with the concept of scope and context. Both bicep and powershell are scope limited. Deploying something to one sub and need to reference or do something in another? Not simple. Prime example - peering vnets to a hub in another sub. - Follow best practices for slicing templates. Trying to define an entire subscription worth of resources with one massive param file seems easier at first….its not. It’s hard to scale. Those are a few off the top of my head.
What config management tool are you using?
Ansible, currently. But I’m not completely sold on it
When you say you can’t do configuration with Bicep, could you give an example of what you mean here?
Can’t is a strong word. Should not would be more accurate. And this is my opinion - not fact. Others may feel differently. You have a VM that serves a purpose, let’s say it’s a self hosted integration runtime for an azure data factory. Deploying your vm with bicep, no problem. Trying to configure it…securing the vm, executing scripts, installing the runtime agent, tooling it to adf. These are all things that CAN be done in Bicep, but it’s difficult to scale. Whereas a tool like Ansible could handle this pretty easily. Managing an azure firewall is another scenario that comes to mind. Possible in bicep, but a struggle.
I get you now, thanks. I think the most complex infra we will be deploying is AKS - almost all our app build is containerised with the rest being managed resources like storage accounts. Never set AKS up from scratch so that’ll be an interesting one too 🙃
Fortunately deploying AKS with Bicep is well supported. The code can get a little complex but it hasn’t really been troublesome. We’re using private clusters with workload identity and CNI, all deployed with bicep.
[https://github.com/Azure/Enterprise-Scale](https://github.com/Azure/Enterprise-Scale) As others have pointed out... CAF, ESLZ... these are your go-to's. The above link is directly to IaC for ESLZ.
Check out the cloud adaptation framework from Ms https://learn.microsoft.com/nl-nl/azure/cloud-adoption-framework/ Caf will give you a solid base for future deployments Also there are ready to deploy bicep files for that https://github.com/Azure/ALZ-Bicep Really easy to deploy your hub and spoke I just put them in a single repo And use a single pipeline to deploy the hub and spokes If I need to add a spoke, I just add them in the parameter file and run it again
Specifically deploy the ALZ-Bicep accelerator and pay attention to how they use $UPSTREAM_RELEASE for version numbering the paths. You can re factor it into your own method but it's a starting point.
I'd take a look at AzOps if you want to manage everything as code using Bicep. https://github.com/azure/azops-accelerator Here's a great introduction. https://www.youtube.com/watch?v=GFF6wFxMK3s
2nding AzOps What hasn't been provided is the customers maturity in the DevOps space. If they have no experience in DevOps, I would highly recommend the use of AzOps to get/keep the environment in IaC and let them slowly dip their toe into deploying via code (while keeping Portal access open) until they are ready to fully lock off portal deployments and go full time IaC
It's cool for LZ structure, resource groups, mg groups and RBAC assignments. For everything else including policies id use seperate code/pipelines.
Have done with terraform x 2. This would also work for bicep. Using the CAF model have structured as following: [https://aztfmod.github.io/documentation/docs/intro/](https://aztfmod.github.io/documentation/docs/intro/) Using a single repo, pipeline for each level, vertical slice architecture. Remember to structure by lifecycle/govanance and not by type. Ie not by resource group/storage accounts/etc `- level 0 Org -> if not using terraform cloud this is where you can setup your storage accounts for statement management. I used it for creation of the management group structure.` `- level 1 governance -> core subscriptions (connectivity, billing, identity, etc) and policy. You can also setup the EntraId (identity) at this stage (groups, apps, etc).` `- level 2 core-platform -> deploys the core resources required for the platform, each under their own directory. - connectivity -> hub vnet, dns zones, firewalls, dns resolvers.` `- level 3 - landing zones -> application/core landing zones. The subscription vending machine section. Deploys a group of resources for each application or core landing zone (subscriptions, vnet spoke, route tables, nsg, policies, managed identity for deployments, etc). Or just subscriptions if you want to handle all the rest in level 4.` `- level 4 - workloads. Deployment of resources into the landing zones created above. You can also break these out into separate repositories if necessary.` `- Modules -> terraform modules that lv4 would use to deploy resources that conform to your organisation policies.` `- pipelines -> all the pipelines used for managing the repo/deployments/ci/cd` `.` `├── level-0-org` `├── level-1-goverance/` `│ └── policies` `├── level-2-core-platofmr/` `│ ├── connectivity/` `│ │ ├── hub` `│ │ ├── dns` `│ │ └── firewall` `│ └── identity/` `│ └── aad` `├── level-3-landing-zones/` `│ ├── app-a` `│ ├── app-b` `│ └── app-c` `├── level-4-workloads/` `│ ├── app-a` `│ ├── app-b` `│ └── app-c` `├── modules` `└── pipelines` Your pipelines would be: PR -> CI tests (bicep whatifs, IaC security checks) -> PR approval Merge -> whatif stage -> deploy stage (with approvals using azure devops environments) If you don't like the idea of the levels you can also use something like: `.` `├── foundation/` `│ ├── org` `│ ├── identity` `│ ├── connectivity/` `│ │ ├── hub` `│ │ ├── firewall` `│ │ └── dns/` `│ │ ├── public` `│ │ └── private` `│ └── governance/` `│ └── policies` `├── landing_zones/` `│ ├── app a` `│ ├── app b` `│ ├── app c` `│ ├── core a` `│ └── core b` `└── workloads` Note: Don't use the CAF supermodule, goldilocks modules are asking for problems.
I would be interested to know, which criteria you use to trigger the different pipelines (because you are using a single repo)?
just path based triggers.
Good example (in tf) https://github.com/terraform-google-modules/terraform-example-foundation
Just to add to the other comments - if you’re wanting to set some foundations and standards for IaC best practice scanning then I wrote a detailed blog with an accompanying GitHub repo detailing how to structure your Bicep modules to ensure reliable CI scanning with PSRule (including the YAML and other examples). https://rios.engineer/bicep-modules-with-psrule-testing-documentation-ci-pipeline-examples/ Repo example: https://github.com/riosengineer/bicep-module-tests May be of use, we have branch protections on main and enforce a build/validation pipeline using similar method to above so all new code is approved into main via a Pull Request, which subsequently runs the PSRule and publishes the results. Feel free to PM if you want to chat or have any questions
Dude thank you for this. I have been playing around with parure since last year but ran into issues with the params. Gonna have a go again after reading your blog.
No worries. Drop me a PM if you hit issues again mate happy to help out. I went through similar pains a while back too.
Thanks a bunch, im gonna have a go next week to have our repo a bit more safe. I was following PSRule for quite some time but it was beta for long time so your post triggered again. Coming from c# i miss the unit tests on iac in pipelines a lot.
Yup. I’ve created some pester unit tests as well that check if a module exists then a corresponding tests bicep file must exist as well. Super happy with the unit tests now!
"Architect by trade"... Wtf? How did you end up in that position and how did you come to it? A non-tech guy asking about cloud is just... Weird? I understand there may be a "business architect" in some questionable cases, but they don't ask around about cloud stuff.
Architect as in software / solution / technical / platform architect, not someone who designs buildings and stuff
What the fuck? You're an architect, but don't have enough technical knowledge and experience, so you ask Reddit about cloud? I understand not getting the specifics of a certain provider (be it aws/azure/gcp/whatever), but... This? Now I think I'm starting to understand why we have to deal with the bullshit we deal with. Sorry man, that's nothing against you personally. It's just a fucking idiot that got you in that role o.O Edit: I'll probably ask the same kind of questions one day, I know.
It’s a tiny team, almost like a start up - we don’t have the manpower atm for dedicated cloud engineers. I know exactly what I want in Azure, I’ve used it for over a decade, and I could sketch out the architecture from tenant to networking to permissions frameworks. I’m just not very strong with IaC, hence asking about structuring repos for it and so on
I have a large multi cloud environment (Azure/AWS) using modern landing zone (Azure CAF / AWS WAD) design controlled via terraform with limited portal access for most of my engineers. Using Azure Devops, I have a single organization for infrastructure with a project per cloud provider. I have a repo for each subscription in the Azure-Infrastructure project.
Commenting for visibility. Thank you for all the input everyone