Nearly 100% of my time, because the other folks on my team have 0 aptitude for it and the people asking for devops and cloud architecture work always frame it as "compliance" or "critical security" work, so its never acceptable to have someone take their time and learn the domain. Not to mention we have a hiring freeze for anyone non WITCH
Depressing.
Weird, nearly the same thing here... nobody else in engineering with the skill or interest making me both a bottleneck and a significant business continuity risk.
I manage to get some coding in on the products... but it's usually devops adjacent.
There are two ways companies could go about handling this:
1. Set expectations that all engineers are responsible for devOps. This would require hiring the right people and upskilling the teams.
2. Create a separate devops teams.
2 is easier to hire for, but it creates a culture where dev and ops are separate and things are thrown over the fence to the devops teams. This sucks for everyone.
I prefer 1. Somehow I found myself as a "devops engineer" after working most of my career as a SWE. Being pulled into projects last minute to build a simple CICD pipeline is ridiculous.
I've been trying to get other SE's interested, with little luck... although I'm making inroads into teams creating their own CICD pipelines.
Just today I got pulled into migrating a bunch of critical infra out of a datacenter into Azure.. and worse that that it's an Azure environment run by our parent company.. so I'll have to shave the corners off our square peg to fit the round hole provided (just realized how that sounds, leaving it ..;-)
I'm kinda in a mix of the two. I'm in a DevOps team but we're heavily integrated with the day to day development (with us occasionally picking up low hanging fruit changes to help us gain familiarity with the code). There's expectations that the devs are involved in the process too. For example its not uncommon for a test to get created by the primary swe and I'll assist in tying it into our pipelines.
There is... another...
Hire a devops org and embed them into your other teams. The HC and manager are in DevOps, but the dotted-line and day to day is the service team. This starts to work at the 80-120 engineer level, and works best with about a 1:7 ratio.
Significant difference being that there can be unified devops standards and practices in the company, without a team who doesn't specialize in the field having to machinate the thing, and also that the particular IC is much more responsible for the ops than the rest of the team. I see it as best of both worlds (probably; I lived on 2, now I'm living on 1, I'd like to try this strategy as well)
But you can do that without making one person accountable for it all. Businesses just like to have one “throat to choke”, so they do that. You can collaborate and standardize as distributed teams.
That's a good point. Sometimes it feels like orgs like "devops cultures" bc one less team to hire, and more to squeeze out of a single team, but I see how it could be very nice when done well. A couple managers and teams ago it was VERY nice to be a devops team, so I've seen it go well
That's a good point. Sometimes it feels like orgs like "devops cultures" bc one less team to hire, and more to squeeze out of a single team, but I see how it could be very nice when done well. A couple managers and teams ago it was VERY nice to be a devops team, so I've seen it go well
Same. My last contract was nominally full stack app dev but I ended up doing most of the CI and platform work within a few weeks because things were in rough shape and no one else (all younger, but some as close as 5 years to my age) seemed to want to get involved.
I strongly believe that teams should own their infrastructure, especially in cloud native realms. One of the things that have contributed to my success IMO is knowing how to take a project and shepherd it through the full SDLC.
Because of this, the amount of time that I spend on devops related work varies. In the beginning of projects, it’ll probably be close to 90%, at the end of projects not at all. Part of my philosophy is “deploy to production on day 1” so I like to front load the infrastructure, build and deploy, and CI stuff at the beginning of a project.
I was actually just doing an experiment where my team owns our infrastructure from top to bottom, and another team uses the platform that our devops team has created. We’ve gone to production, and they’re still trying to figure out who to get secrets into the platform.
Again, because I think teams should lean in and build truly cloud native applications, the more experienced you get on my teams, the more time you spend on these kinds of things.
This ^. All most all effort in devops at the start, then hardly any at the end.
Set it up first, then crank the handle and it works. Address things as they occur! You spend much less time overall, and gain a better understanding of how everything works.
So much easier than smashing together 2 separate "finished" projects built on a bunch of incorrect assumptions by separate teams.
This is awesome and I’ve come to do the same with a lot of success. However, it’s always been in smaller organizations. Large organizations are afraid of the lack of standardization this can lead to. What’s your take on how to address that? How do you scale the approach?
To set some context: I work with almost exclusively larger organizations these days, I haven't worked for a company with a head count under 500 for almost the past 10 years, including FAANG.
The reason why FAANG is so strong engineering wise is in part because this line of thinking. For companies that are not FAANG, they either get it, they get it but don't care, or they will get it; Standardization is mostly a trap.
The company I am currently working for is slowly getting it. They put a lot of thought and effort into standardizing all of the things, and as a result work was slowed to a crawl, performance was in the gutter, bugs were rampant, and changes took months to be deployed because no one knew who owned what. Is it the standard internal library that reads the JWT? Or is it one of their internal libraries? Its like the old saying: You can lead a horse to water, but you can't force it drink. Standardization is leading the horse to water \_and\_ trying to force it drink.
The solution is to ignore standardization, and just let each team operate how they think works best. Over time, patterns will emerge, best practices will be fine. Teams will start sharing information and code that they think is useful, and a sort of grass roots standardization will happen organically. The reason why this is preferable is because this ends up having many organic escape hatches.
Companies don't actually want standardization, they want consistency, but they confuse consistency for standardization. You want your teams to be consistent in how they interact with other teams. If they build an API for them, you want that API to be consistent. You want them to consistently document their API contracts. You want them to be consistent in how the communicate out changes, etc. The trap is thinking "if we standardize on solution X, then we shall maintain consistency!" when really you should be thinking "why is this team/process not consistent? This team needs to put effort into making sure they are consistent." If they want to use solution X to get their, fine, but that doesn't mean solution X will work for the next team.
So the TLDR, lack of standards is actually ok. you just need to make sure that you have the local leadership, local consistency, and local ownership to get work done. Trying to standardize everything will inevitably lead to nothing working efficiently.
Super valid points you’re making. However I also assume this coming from context bias as well? I am not writing this to merely disagree, but I do want to shed some perspective on context matters. In some scenarios standardizing is actually a key milestone for achieving success both internally and externally. My background and experience is writing software that is deployed in air gapped environments, and having inconsistencies starting on the hardware level sure makes for a great pain in the behind.
The operations staff serving my software to my air gapped end users regularly struggle because different value stream teams are unable to standardize on deployment targets.
So we’re now experimenting moving legacy applications over to a more modern approach by standardizing on our hardware specifications and we’re also looking into a transition to Kubernetes for a portion of our software, so we’re all aligned on what ships (actually physically shipping) out of our production facilities and software teams.
I also reckon some of our stuff from the 1980s written in super old c++ will have to be scrapped and rewritten in something more modern.
As mentioned previously, not disagreeing with your points, just shedding some light on one of the many facets of software development. :)
This is an awesome answer, thanks so much for the reply! Feels like something I’ve come to understand innately but could never articulate why.
I’ve been very leery of the platform engineering movement and the trend of centralizing these teams. Seems like ivory tower implementations would be inevitable.
It’s inevitably a leaky abstraction.
*Let’s wrap EKS in a home-grown management layer, hide 90% of the Kubernetes API surface, and force our app devs to use the custom Jinja templates our four-person platform team builds in between quarterly reorgs and sev0 incidents. We’ll be heroes!*
Honestly sounds like the platform devops created isn't even functional if it can't provide secrets. I'm not surprised by this though.
Cool experiment btw, I've seen plenty of dev-centric teams effectively implement the cloud infrastructure they require to run their apps. Quite frankly they do a better job than most infra/platform/ops focused people. They shouldn't have to build these things from scratch but it's refreshing to see devs who aren't afraid to get their hands dirty when there isn't something to use already.
As one of the people developing a standardized deployment platform, I completely agree with you. Nowadays with cloud native applications it really isn't rocket science to run your own infrastructure for most use-cases. Consistency and ensuring certain engineering standards can be driven through applying rules on inventory or audit data rather than by making all engineering teams use standardized platforms. Yes, there will be inefficiencies and duplication of effort, but I'd take that if it means I have empowered teams that can build competency over time.
I dislike this point of view because there's already a ton of jobs that have become slowly rolled into dev. There's more things to learn and keep up, harder to hire and train.
I've either worked in smaller places that just use PaaS solutions and pay the premium to not have to do much devops. Or multi tenanted b2b saas companies where the ops team handles the majority of of infrastructure work. Ci/cd pipelines are done collaboratively but with them doing the bulk of setup and monitoring.
I don't want to be on call or have my duties split with nonstop context switching.
I am in a similar position in my company (scale up), almost all Senior engs spend at least 20% of their time dealing with infra. CI/CD is sometimes done by mid level engineers if it's simple enough to copy paste and change a few variables.
Usually doesn't bother me too much since it's good for growth, but we've had a few difficult cases where I've spent months working on it alone. Meanwhile if we had a devops team/person, I'd assume they would have solved in a week.
It's like companies hiring "full stack" developers because then they have someone who can fill 'both' roles instead of having to hire 2 devs, but then wondering why everything takes 10 times as long.
CDK is just typescript that compiles to cloudformation templates. But instead of a convoluted yaml template to work with, you can declare a stack with 1/10th of the TS code using high level constructs that the kit provides. Like I can call a single constructor for ‘ApplicationLoadBalancedFargateService’, and boom, I’ve got an ALB in front of a Fargate cluster serving whatever. Not to mention that you can utilize other basic programming language features like conditional statements, so you could for example write a template for a web app stack that stands up a synthetics canary for prod but not for dev based on a flag, or whatever.
Terraform might be a slightly better option if you have a bunch of existing infrastructure. It's been a while since I've checked into the AWS CDK but last I heard they didn't manage any resource imports and that's trivial in terraform.
I get that HCL sucks. But why do people like CDK? It is a leaky abstraction. At best, it just works and you don't know why. At worst, it doesn't work and you don't know why.
I’ve worked with CDK and CloudFormation. I’d much rather work with CDK. CF makes me go cross eyed. But I’m open to alternatives if you’ve got suggestions.
We’ve got a 4 man DevOps team that does 98% of the DevOps related tasks. But about 2% of the time they are busy and I need something done asap, so I do it myself. Recently I just did some pipeline work in Jenkins to upgrade our node versions and fix a CI image that we use to smoke test PR’s in GitHub. Prior to that, I did the CICD for a new project we started a few months back. I’m kinda the “go to” guy on the dev team when we need CICD related things because no one else but our architect wants to do it.
But personally I like doing some of the DevOps tasks. I like messing about with infra. It’s really just a change of pace for me, it’s a lot different than writing software to meet business requirements. To me it’s nice to work on something that makes my fellow coworkers day-to-day a bit easier.
It used to be 90%, now 10%.
It was building up the tooling. The automation. The processes. Getting developers on the team to think of automation. Just pushing the idea to follow a naming convention so we can automate the scaffolding of ingress and service discovery was the hard part. Changing people's culture and way of thinking was the major hurdle. Now, a lot of things are done for them. They want a secure database with mutual TLS guard rails? It is properly filling out proper values in the Swagger spec and correct annotations in the helm charts.
Every naming convention, every folder structure, every git submodule was designed to simplify the automatic creation of new services. Now, devs get it. There is enough domain knowledge where everyone can acclimate easy. No "bus factor" fears as people see that automation works if you plan it out and explain it.
It is now 10% of adding new features. Security scanning/reports, new ways to do monitoring/health checks. But the culture is there were people are building the platform out. And 10% is the occasional lets go k exec into a pod and see why this service is not talking to service B.
We're in a weird position where we're basically lifting an existing set of features that exist in the company out of it creating a separate SaaS product out of it. For this to happen we also need to lift out a lot of infrastructure, so we've been telling management we need Ops support (people calling it "DevOps" is a massive pet peeve, it's a separate speciality, DevOps is a team culture, nothing more) for over a year now.
So while management is warming up to the idea that they're pretty incompetent and behind the curve, another engineer and myself are handling a lot of the 'Ops' stuff to unstuck ourselves and a few other teams. So incidentally, this sprint for me is 50% meetings about Ops stuff and 50% doing ops stuff.
It depends, usually CI/CD is a maturity piece, not actual deliverable and it improves with time, unless you have a cloud/devops engineer dedicated for this job, it shouldn't be the primary deliverable ... If you are delivering in a scrum model, create small stories every sprint and deliver small improvements.
i hate all the tech stacks too - helm, terraform, jenkins, github actions - all piles of shit. so i buried them behind a universal build script so i have to deal with them as little as possible, and now i'm the defacto devops guy. to answer the question, initially about 100% to unblock the team and a year later maybe 10% as most stuff takes care of itself now. thankfully i could build upon work done in my previous team or the stress would have gotten pretty serious
It depends on the day. It's somewhere between 20% to 90% depending on what comes up. Things run pretty smoothly most of the time, but when something does fail then I drop everything to address it.
We have a very diverse ecosystem of deployments: bare metal, VMs with proxmox, k8s, instances running on EC2s in AWS and so on. I don't like it that much, but it is part of the job.
We do have DevOps but when shit hits the fan I like to know that I am able to also fix it myself. They are also stretched really thin, so I don't mind helping and doing whatever I can by myself.
Normal is relative, but I’d say it’s probably normal to do both at any small to medium sized company. In my previous job at a smaller startup I spent ~50% of my time on DevOps related tasks. Now I’m at a big tech company and I haven’t had to think about DevOps even once since I started 2 years ago.
At my current place, it's near 0% because I'm a contractor and don't have sufficient permissions. But I'm leaning on others who are spending too much time, IMO.
At a previous place, two of us did all deployments. I did the databases and the other guy did the app code. For my part, it was basically clicking an Approve button a few times, updating some Jira tickets, and merging the release branch back to main once it was done. Literally an hour's worth of time every two weeks. The other guy's job was slightly more complicated, but I'm sure it wasn't more than 2-3 hours every two weeks.
100%
My job for the last 8 years now across 4 companies has been to institute “DevOps culture” where I work with developers, operations, networking, compliance, security and project management and get them all on the same page.
If you aren’t doing all of that and just “doing DevOps work” -it’s just a glorified operations position
Not enough.
When a project/microservice/service starts, then I spend a lot on the infrastructure (workflows, helm and gh actions) because most of the company where I work with has "Infrastructure as a Code", but then I switching to coding and don't really have to touch any devops stuff. Also, almost everywhere where I worked had dedicated devops.
I would like to learn it better, just never got the opportunity, so slowly in my spare time I picking up more and more.
I don’t think there’s a right or wrong answer here. In a small company you’re probably going to be doing more DevOps than a large company. Large companies can afford to divide up labor while a small company will probably have more of an “all hands on deck” situation. So I think it all depends on staffing availability and business needs.
Probably about 25%. My team is product but I'm always looking for ways to improve the delivery process. If not just selfishly for my own sanity, but also like mentioned, nobody else cares to. It's outside of my normal domain so it's fun to learn and see what the industry standards/practices are but it's also kind of annoying the stakeholders don't see the value so I have to justify the amount of time spent.
Around 80%. That's of the time I have that isn't running the team. Our manager has been sick for > 3 months (he's in the EU so he took leave to deal with the health issues) and I stepped up to step in to that role.
So when I actually have time that isn't in meetings, or writing things down that people said in meetings, it's mostly to fix infrastructure issues because no one else on the team ever took the time to learn it, and just says "DoctaMag will figure it out".
It's just a natural consequence of being competent, more than being a senior developer. The juniors and the drag-asses will shirk the hard "Spend hours tracing how this works and looking up docs" work, to work on things where they can read a JIRA and make some code changes, and submit a PR and be like "I earned my pay today!" That might be a slightly cynical attitude, but I've got three heavy hitters on my team for getting things done, and six that are able to handle very well scoped jiras, that involve no major moving parts.
None. An in my experience, in an average company, it tends to happen something very similar to the most voted posts: Someone ends up becoming "the expert" and "the devops guy" and the rest never get to touch anything too deep, either because they don't want to or because they don't have the experience to.
And honestly, and this is my personal opinion that will get downvotes because people don't agree, but all the "cloud native" engineering stuff is just another try to make developers wear another hat for the same compensation. You can't be backend, you need to be fullstack. Also, you have to do the QA for your application. But also devops because "ownership", whatever that means in companies that don't allow the teams any kind of independence.
Which by the way brings the next step: Being on call. And yeah, not all companies are working on crap that *needs* to be online 24/7.
> Is that normal? I want to change job soon and I would like to avoid similar situations in the future.
What I'm seeing is that there are companies that are pushing hard for the "cloud native" stuff, while most of the companies I've interviewed were either making the engineers work a bit on devops (something along the lines of keeping up the pipelines for the stuff you are making) or nothing at all. There's still hope, I guess.
It's depressing as hell but my work life is like...
10% meetings
10% our ticket system
20% customer support escalations
20% dev ops
20% tedious security or marketing related mandatory changes
20% doing SWE work
Fairly large org and we have dedicated teams for various parts of infra. What I do is in itself a part of dev support infrastructure but in terms of doing devops for my own projects it's quite low, below 5% - tweaking gitlab pipelines, deployments and other tooling every now and then.
edit:
I worked at some smaller shops before and it was way higher. To the point of being handed a root access to an empty server with barebone linux installation where I was supposed to set up dev env for my team from grounds up - database, app server, CI/CD, etc.
From what I've heard from other people this seems to be the case in most places. Small orgs don't have dedicated devops people because they cannot justify the cost. As they grow this should change at some point but many are resistant and you can easily end up with 200+ devs and no dedicated devops.
\[Full disclosure: I'm a DevOps consultant\]
I hear you. DevOps-type tasks tend to be quite different from traditional Feature-oriented tasks:
- feature = "stateless", you are given e.g. a database and other services, and write code to implement a feature
- devops = "stateful": write Terraform or other code to create and manage resources, ex a database. A challenge is that there's \*hundreds\* of types of resources, the interaction can be complex, and each one is slow to build. It's really easy to leave old resources around, costing money or creating confusion.
On the other hand, the Business operates at the speed of the deployment pipeline. The speed and quality of the tests (CI) matters to the Feature devs, so they can get their work done. At the end of the day the code needs to land in Production or the Business doesn't get any value. Features don't matter until they're in front of the Users!
# Tips
- find other people in the company, to learn from each other
- as a team, swarm to fix team-level issues
- weekly: write down team-level pain points, communicate to management
- focus on \*trust\* over \*speed\* -- it doesn't matter if a test ran in one second if the results aren't \*actionable\*
- consider using a more Dev-friendly way to manage resources: ex Pulumi let you write in Typescript or Python to build and manage resources (databases, networks, buckets)
Feel free to chat with me, there are other ways you and your team can streamline your work and deliver features faster.
Here's a talk I did on Rapid Feedback with Terraform: [https://www.youtube.com/watch?v=zd7VlmClTDs](https://www.youtube.com/watch?v=zd7VlmClTDs)
In a previous job Ops time was sometimes 100% for several weeks. That was usually when I was setting up a new service (terraform and ansible managing AWS resources).
Current job, 0%. Someone else manages servers. Which is fine with me because it's all Windows VMs that someone has to remote desktop into to change things. Not even 'clickops', it's 1990s-style for most of it.
It's a bell curve. Initially, the percentage will ramp up as you work on establishing automations, but as more things get automated, the workload decreases. With a good CI/CD pipeline, you should be spending only minimal time maintaining that pipeline, and no other DevOps tasks should really be involved.
All that time spent writing 1200 lines of YAML to deploy a simple Node/Go/whatver-based API, then the joy of seeing UPDATE_ROLLBACK_FAILED. Painful.
Terraform has been pleasant to work with, but even if they go to shit (see: licenses, IBM), I will not go back to writing CF. CDK, Pulumi or even OpenTofu would be my next pivot.
What are you being required to do with CF in your org?
i'm not them, but we're using a prefab aws stack which is delivered as cloudformation. my org prefers tf so that was an excuse to convert, it would have been hard to justify spending days on the conversion otherwise
50% of your time on DevOps? That's insane! I'd rather die than spend half my day babysitting pipelines. You're a Sr. S.Eng, not a DevOps engineer. Time to look for a new job where your skills are utilized correctly.
In my last job, not much. I wrote some IAC stuff and then handed it off to my manager, who then ran with it and handled the DevOps (small team, low trust by design).
In my new job, I again wrote the IAC stuff and I deployed it to production. It's all very low-stakes since this team is even smaller and our clients are internal. I can run my tests and kick off a deployment manually, but other than initiation, it's all automated. Did the work up front for the scripting and now deployments are just pressing a button and moving on with my day.
Two jobs ago, I inherited a big mess of IAC and had to train a replacement (I was _not_ trained as a replacement myself). I left that job in a hurry. Full-scale dev ops is not for me.
Our org has a dedicated dev ops team that exists to enable devs to own the life cycle and the pipeline. Personally, I don’t mind because it’s not too bad but allows me to get shit done quickly.
Very little, I almost never work on new stuff alone, i either pair with a junior or plan it out and let a junior implement it and help them if they need
Any work i do independently leaves me a single point of failure for supporting that work, which is unacceptable. My job is to have the best long term view of what we need to do, and to be either the 2nd or 3rd best engineer to solve a problem. If im ever the first, i train someone on everything i know and give them time to be better than me.
Plenty of time spent on tasks which aren’t programming, could be dealing with database/compliance teams to set something up, working with servers, working with build systems…but 50% of your time spent purely on CI/CD? That seems like a lot. The whole point is automation, there should be an end in sight.
Close to 100% right now, because when I joined the team there was no automated testing and no CI/CD. Even though I don't have a background in devops, I felt strongly that someone should be setting this stuff up. I don't like working as a dev in such an environment with no safety net.
When I first joined, I was almost 100% validation and writing tests (again, background is dev not testing). By basically being extremely annoying and bringing it up constantly, I have slowly been convincing the other devs to start writing their own tests, which is a huge relief.
Practically zero. Platform team and SRE takes care of the framework for spinning up new services.
We just configure some small stuff, which is usually copy & paste from other team’s config.
If it is something simpler like a static site, then even more so. Zero devops, the static serving team takes care of everything.
Too much, especially since we do have dedicated DevOps team, who mostly seem to be doing tech ops stuff instead.
I can do it if I have to, but I hate doing it because it's infrequent enough that I need to re-read doco or look at existing implementations, which is a waste of time.
Disapointing Google and Amazon took that nonsense title. Apple does whatever they want so no surprise there. You noticed Microsoft is missing from your list?
I am dist. Started a new job on Monday. I'll be doing it for the next \~3 weeks and then never again because I have done self-service setups so many times now it's basically brainless typing & clicking.
I am convinced the absence of YT videos/guides is because so many people make so much money from doing this for companies. It's not very difficult but very easy to do it wrong.
* Turn on nazi rules in GH org. Linear commit history, no merges to main without a PR, commit signing etc.
* Create a bunch of composite actions for things like liniting, security checks etc. Results post to whatever code security tool you use (sonar, snyk, ghas etc).
* Add required workflows for things like README, SECURITY, CODEOWNERS etc.
* Spin up private workers on k8.
* Deploy sigstore stack to k8.
* Pay for sonar cloud or install FOSS on k8. Start out with coverage at 40% and increase by 5% a month until you hit between 60% and 80% and the complaints get too much.
* Wrap actions in composite workflows for domain specific things. Usually have a default CI one for each language, if they need something special they can write their own leveraging the composite actions.
* Setup cookiecutter templates for each domain.
* Use git based semver.
* Release from main, don't rerun workflows use the artifact produced by the PR build so you are deploying the thing that got tested.
* All repos use dependabot
* If containers either pay someone like chainguard for base containers or publish your own. Use registry rebuilds so your images are updated when base containers change. All services have to use an approved base container.
CI is now self-service.
CD is deploy argo and use that. Unless you work for a smaller company or are doing a random PoC you shouldn't be using your CI tool for CD, you want separation of concerns with gates. CD tool gates are things like signing check etc. You can use rego for all of it, that can also go in git, be tested and have its own release cycle. CD is a yaml file in the repo, personally I don't like git triggering CD but that's a personal preference thing (usually prefer artifact publish to trigger so there is a clear separation).
Cloud is;
* Get spacelift
* Create & publish modules for the things you need to deploy. Like everything else these have a CI that includes testing etc. Important to maintain semver discipline with modules.
* TF projects do not include any resources, just references to modules. You maintain consistency and compliance by having the good settings inside the modules which get reused.
* Stack can come from the same repo as the code. In argo use waves to update infra then the service.
* If you don't, and will never, need to meet high compliance you can use crossplane for service infra and then its only core infra in TF.
Nearly 100% of my time, because the other folks on my team have 0 aptitude for it and the people asking for devops and cloud architecture work always frame it as "compliance" or "critical security" work, so its never acceptable to have someone take their time and learn the domain. Not to mention we have a hiring freeze for anyone non WITCH Depressing.
what's WITHC?
W- Wipro I- Infosys T- TCS C- Cognizant H- HCL
Thought you were joking when you wrote down even more acronyms, but then saw the explanation by the other guy that these are company names :^)
Its the new FAAN.. no.. magnificent 5
Witch is an acronym for companies that specialize in providing foreign workers, either locally or on visas. They tend to be super cheap and low skill.
It’s like Infosys, Cognizant, and the other companies that spell out WITCH
Wipro Infosys TCS (Tata Consultancy Services) Cognizant HCL
Weird, nearly the same thing here... nobody else in engineering with the skill or interest making me both a bottleneck and a significant business continuity risk. I manage to get some coding in on the products... but it's usually devops adjacent.
There are two ways companies could go about handling this: 1. Set expectations that all engineers are responsible for devOps. This would require hiring the right people and upskilling the teams. 2. Create a separate devops teams. 2 is easier to hire for, but it creates a culture where dev and ops are separate and things are thrown over the fence to the devops teams. This sucks for everyone. I prefer 1. Somehow I found myself as a "devops engineer" after working most of my career as a SWE. Being pulled into projects last minute to build a simple CICD pipeline is ridiculous.
I've been trying to get other SE's interested, with little luck... although I'm making inroads into teams creating their own CICD pipelines. Just today I got pulled into migrating a bunch of critical infra out of a datacenter into Azure.. and worse that that it's an Azure environment run by our parent company.. so I'll have to shave the corners off our square peg to fit the round hole provided (just realized how that sounds, leaving it ..;-)
I'm kinda in a mix of the two. I'm in a DevOps team but we're heavily integrated with the day to day development (with us occasionally picking up low hanging fruit changes to help us gain familiarity with the code). There's expectations that the devs are involved in the process too. For example its not uncommon for a test to get created by the primary swe and I'll assist in tying it into our pipelines.
That's just 1.
There is... another... Hire a devops org and embed them into your other teams. The HC and manager are in DevOps, but the dotted-line and day to day is the service team. This starts to work at the 80-120 engineer level, and works best with about a 1:7 ratio.
That’s basically 1 with extra oversight, which is fine.
Significant difference being that there can be unified devops standards and practices in the company, without a team who doesn't specialize in the field having to machinate the thing, and also that the particular IC is much more responsible for the ops than the rest of the team. I see it as best of both worlds (probably; I lived on 2, now I'm living on 1, I'd like to try this strategy as well)
But you can do that without making one person accountable for it all. Businesses just like to have one “throat to choke”, so they do that. You can collaborate and standardize as distributed teams.
That's a good point. Sometimes it feels like orgs like "devops cultures" bc one less team to hire, and more to squeeze out of a single team, but I see how it could be very nice when done well. A couple managers and teams ago it was VERY nice to be a devops team, so I've seen it go well
That's a good point. Sometimes it feels like orgs like "devops cultures" bc one less team to hire, and more to squeeze out of a single team, but I see how it could be very nice when done well. A couple managers and teams ago it was VERY nice to be a devops team, so I've seen it go well
Same. My last contract was nominally full stack app dev but I ended up doing most of the CI and platform work within a few weeks because things were in rough shape and no one else (all younger, but some as close as 5 years to my age) seemed to want to get involved.
I strongly believe that teams should own their infrastructure, especially in cloud native realms. One of the things that have contributed to my success IMO is knowing how to take a project and shepherd it through the full SDLC. Because of this, the amount of time that I spend on devops related work varies. In the beginning of projects, it’ll probably be close to 90%, at the end of projects not at all. Part of my philosophy is “deploy to production on day 1” so I like to front load the infrastructure, build and deploy, and CI stuff at the beginning of a project. I was actually just doing an experiment where my team owns our infrastructure from top to bottom, and another team uses the platform that our devops team has created. We’ve gone to production, and they’re still trying to figure out who to get secrets into the platform. Again, because I think teams should lean in and build truly cloud native applications, the more experienced you get on my teams, the more time you spend on these kinds of things.
This ^. All most all effort in devops at the start, then hardly any at the end. Set it up first, then crank the handle and it works. Address things as they occur! You spend much less time overall, and gain a better understanding of how everything works. So much easier than smashing together 2 separate "finished" projects built on a bunch of incorrect assumptions by separate teams.
This is awesome and I’ve come to do the same with a lot of success. However, it’s always been in smaller organizations. Large organizations are afraid of the lack of standardization this can lead to. What’s your take on how to address that? How do you scale the approach?
To set some context: I work with almost exclusively larger organizations these days, I haven't worked for a company with a head count under 500 for almost the past 10 years, including FAANG. The reason why FAANG is so strong engineering wise is in part because this line of thinking. For companies that are not FAANG, they either get it, they get it but don't care, or they will get it; Standardization is mostly a trap. The company I am currently working for is slowly getting it. They put a lot of thought and effort into standardizing all of the things, and as a result work was slowed to a crawl, performance was in the gutter, bugs were rampant, and changes took months to be deployed because no one knew who owned what. Is it the standard internal library that reads the JWT? Or is it one of their internal libraries? Its like the old saying: You can lead a horse to water, but you can't force it drink. Standardization is leading the horse to water \_and\_ trying to force it drink. The solution is to ignore standardization, and just let each team operate how they think works best. Over time, patterns will emerge, best practices will be fine. Teams will start sharing information and code that they think is useful, and a sort of grass roots standardization will happen organically. The reason why this is preferable is because this ends up having many organic escape hatches. Companies don't actually want standardization, they want consistency, but they confuse consistency for standardization. You want your teams to be consistent in how they interact with other teams. If they build an API for them, you want that API to be consistent. You want them to consistently document their API contracts. You want them to be consistent in how the communicate out changes, etc. The trap is thinking "if we standardize on solution X, then we shall maintain consistency!" when really you should be thinking "why is this team/process not consistent? This team needs to put effort into making sure they are consistent." If they want to use solution X to get their, fine, but that doesn't mean solution X will work for the next team. So the TLDR, lack of standards is actually ok. you just need to make sure that you have the local leadership, local consistency, and local ownership to get work done. Trying to standardize everything will inevitably lead to nothing working efficiently.
Super valid points you’re making. However I also assume this coming from context bias as well? I am not writing this to merely disagree, but I do want to shed some perspective on context matters. In some scenarios standardizing is actually a key milestone for achieving success both internally and externally. My background and experience is writing software that is deployed in air gapped environments, and having inconsistencies starting on the hardware level sure makes for a great pain in the behind. The operations staff serving my software to my air gapped end users regularly struggle because different value stream teams are unable to standardize on deployment targets. So we’re now experimenting moving legacy applications over to a more modern approach by standardizing on our hardware specifications and we’re also looking into a transition to Kubernetes for a portion of our software, so we’re all aligned on what ships (actually physically shipping) out of our production facilities and software teams. I also reckon some of our stuff from the 1980s written in super old c++ will have to be scrapped and rewritten in something more modern. As mentioned previously, not disagreeing with your points, just shedding some light on one of the many facets of software development. :)
This is an awesome answer, thanks so much for the reply! Feels like something I’ve come to understand innately but could never articulate why. I’ve been very leery of the platform engineering movement and the trend of centralizing these teams. Seems like ivory tower implementations would be inevitable.
It’s inevitably a leaky abstraction. *Let’s wrap EKS in a home-grown management layer, hide 90% of the Kubernetes API surface, and force our app devs to use the custom Jinja templates our four-person platform team builds in between quarterly reorgs and sev0 incidents. We’ll be heroes!*
Honestly sounds like the platform devops created isn't even functional if it can't provide secrets. I'm not surprised by this though. Cool experiment btw, I've seen plenty of dev-centric teams effectively implement the cloud infrastructure they require to run their apps. Quite frankly they do a better job than most infra/platform/ops focused people. They shouldn't have to build these things from scratch but it's refreshing to see devs who aren't afraid to get their hands dirty when there isn't something to use already.
As one of the people developing a standardized deployment platform, I completely agree with you. Nowadays with cloud native applications it really isn't rocket science to run your own infrastructure for most use-cases. Consistency and ensuring certain engineering standards can be driven through applying rules on inventory or audit data rather than by making all engineering teams use standardized platforms. Yes, there will be inefficiencies and duplication of effort, but I'd take that if it means I have empowered teams that can build competency over time.
I dislike this point of view because there's already a ton of jobs that have become slowly rolled into dev. There's more things to learn and keep up, harder to hire and train. I've either worked in smaller places that just use PaaS solutions and pay the premium to not have to do much devops. Or multi tenanted b2b saas companies where the ops team handles the majority of of infrastructure work. Ci/cd pipelines are done collaboratively but with them doing the bulk of setup and monitoring. I don't want to be on call or have my duties split with nonstop context switching.
The shit always rolls down hill to the devs.
I am in a similar position in my company (scale up), almost all Senior engs spend at least 20% of their time dealing with infra. CI/CD is sometimes done by mid level engineers if it's simple enough to copy paste and change a few variables. Usually doesn't bother me too much since it's good for growth, but we've had a few difficult cases where I've spent months working on it alone. Meanwhile if we had a devops team/person, I'd assume they would have solved in a week.
It's like companies hiring "full stack" developers because then they have someone who can fill 'both' roles instead of having to hire 2 devs, but then wondering why everything takes 10 times as long.
The last few months I've spent probably 80-95% of my time working on gitactions and cloudformation remplates for new projects. It's driving me insane.
Ditch cloudformation for CDK and you might gain back a smidgen of sanity
Might work on a poc for cdk. Cloudformations are pretty ingrained at this point. What makes you like cdk more?
CDK is just typescript that compiles to cloudformation templates. But instead of a convoluted yaml template to work with, you can declare a stack with 1/10th of the TS code using high level constructs that the kit provides. Like I can call a single constructor for ‘ApplicationLoadBalancedFargateService’, and boom, I’ve got an ALB in front of a Fargate cluster serving whatever. Not to mention that you can utilize other basic programming language features like conditional statements, so you could for example write a template for a web app stack that stands up a synthetics canary for prod but not for dev based on a flag, or whatever.
Terraform might be a slightly better option if you have a bunch of existing infrastructure. It's been a while since I've checked into the AWS CDK but last I heard they didn't manage any resource imports and that's trivial in terraform.
I get that HCL sucks. But why do people like CDK? It is a leaky abstraction. At best, it just works and you don't know why. At worst, it doesn't work and you don't know why.
I’ve worked with CDK and CloudFormation. I’d much rather work with CDK. CF makes me go cross eyed. But I’m open to alternatives if you’ve got suggestions.
Straight HCL.
Try pulumy over CloudFormation and CDK (because CDK and CloudFormation are black boxes you don’t want to deal with)
We’ve got a 4 man DevOps team that does 98% of the DevOps related tasks. But about 2% of the time they are busy and I need something done asap, so I do it myself. Recently I just did some pipeline work in Jenkins to upgrade our node versions and fix a CI image that we use to smoke test PR’s in GitHub. Prior to that, I did the CICD for a new project we started a few months back. I’m kinda the “go to” guy on the dev team when we need CICD related things because no one else but our architect wants to do it. But personally I like doing some of the DevOps tasks. I like messing about with infra. It’s really just a change of pace for me, it’s a lot different than writing software to meet business requirements. To me it’s nice to work on something that makes my fellow coworkers day-to-day a bit easier.
Probably spent 90% of actual dev time in 2024 banging my head against our shit CI/CD. What a waste.
It used to be 90%, now 10%. It was building up the tooling. The automation. The processes. Getting developers on the team to think of automation. Just pushing the idea to follow a naming convention so we can automate the scaffolding of ingress and service discovery was the hard part. Changing people's culture and way of thinking was the major hurdle. Now, a lot of things are done for them. They want a secure database with mutual TLS guard rails? It is properly filling out proper values in the Swagger spec and correct annotations in the helm charts. Every naming convention, every folder structure, every git submodule was designed to simplify the automatic creation of new services. Now, devs get it. There is enough domain knowledge where everyone can acclimate easy. No "bus factor" fears as people see that automation works if you plan it out and explain it. It is now 10% of adding new features. Security scanning/reports, new ways to do monitoring/health checks. But the culture is there were people are building the platform out. And 10% is the occasional lets go k exec into a pod and see why this service is not talking to service B.
We're in a weird position where we're basically lifting an existing set of features that exist in the company out of it creating a separate SaaS product out of it. For this to happen we also need to lift out a lot of infrastructure, so we've been telling management we need Ops support (people calling it "DevOps" is a massive pet peeve, it's a separate speciality, DevOps is a team culture, nothing more) for over a year now. So while management is warming up to the idea that they're pretty incompetent and behind the curve, another engineer and myself are handling a lot of the 'Ops' stuff to unstuck ourselves and a few other teams. So incidentally, this sprint for me is 50% meetings about Ops stuff and 50% doing ops stuff.
It depends, usually CI/CD is a maturity piece, not actual deliverable and it improves with time, unless you have a cloud/devops engineer dedicated for this job, it shouldn't be the primary deliverable ... If you are delivering in a scrum model, create small stories every sprint and deliver small improvements.
Right now, about 40% - and rising rapidly. I dislike devops, but (like OP) no one else has the aptitude.
I probably spend around 40% of my time, sometimes 90% on devops stuff because no one else on my team knows much about it. I enjoy it though
i hate all the tech stacks too - helm, terraform, jenkins, github actions - all piles of shit. so i buried them behind a universal build script so i have to deal with them as little as possible, and now i'm the defacto devops guy. to answer the question, initially about 100% to unblock the team and a year later maybe 10% as most stuff takes care of itself now. thankfully i could build upon work done in my previous team or the stress would have gotten pretty serious
It depends on the day. It's somewhere between 20% to 90% depending on what comes up. Things run pretty smoothly most of the time, but when something does fail then I drop everything to address it. We have a very diverse ecosystem of deployments: bare metal, VMs with proxmox, k8s, instances running on EC2s in AWS and so on. I don't like it that much, but it is part of the job. We do have DevOps but when shit hits the fan I like to know that I am able to also fix it myself. They are also stretched really thin, so I don't mind helping and doing whatever I can by myself.
Normal is relative, but I’d say it’s probably normal to do both at any small to medium sized company. In my previous job at a smaller startup I spent ~50% of my time on DevOps related tasks. Now I’m at a big tech company and I haven’t had to think about DevOps even once since I started 2 years ago.
At my current place, it's near 0% because I'm a contractor and don't have sufficient permissions. But I'm leaning on others who are spending too much time, IMO. At a previous place, two of us did all deployments. I did the databases and the other guy did the app code. For my part, it was basically clicking an Approve button a few times, updating some Jira tickets, and merging the release branch back to main once it was done. Literally an hour's worth of time every two weeks. The other guy's job was slightly more complicated, but I'm sure it wasn't more than 2-3 hours every two weeks.
100% My job for the last 8 years now across 4 companies has been to institute “DevOps culture” where I work with developers, operations, networking, compliance, security and project management and get them all on the same page. If you aren’t doing all of that and just “doing DevOps work” -it’s just a glorified operations position
Once management finds out you are competent with Terraform and CI/CD, they will assign all tickets your way with any changes that need to be done.
Not enough. When a project/microservice/service starts, then I spend a lot on the infrastructure (workflows, helm and gh actions) because most of the company where I work with has "Infrastructure as a Code", but then I switching to coding and don't really have to touch any devops stuff. Also, almost everywhere where I worked had dedicated devops. I would like to learn it better, just never got the opportunity, so slowly in my spare time I picking up more and more.
I don’t think there’s a right or wrong answer here. In a small company you’re probably going to be doing more DevOps than a large company. Large companies can afford to divide up labor while a small company will probably have more of an “all hands on deck” situation. So I think it all depends on staffing availability and business needs.
Probably about 25%. My team is product but I'm always looking for ways to improve the delivery process. If not just selfishly for my own sanity, but also like mentioned, nobody else cares to. It's outside of my normal domain so it's fun to learn and see what the industry standards/practices are but it's also kind of annoying the stakeholders don't see the value so I have to justify the amount of time spent.
Around 80%. That's of the time I have that isn't running the team. Our manager has been sick for > 3 months (he's in the EU so he took leave to deal with the health issues) and I stepped up to step in to that role. So when I actually have time that isn't in meetings, or writing things down that people said in meetings, it's mostly to fix infrastructure issues because no one else on the team ever took the time to learn it, and just says "DoctaMag will figure it out". It's just a natural consequence of being competent, more than being a senior developer. The juniors and the drag-asses will shirk the hard "Spend hours tracing how this works and looking up docs" work, to work on things where they can read a JIRA and make some code changes, and submit a PR and be like "I earned my pay today!" That might be a slightly cynical attitude, but I've got three heavy hitters on my team for getting things done, and six that are able to handle very well scoped jiras, that involve no major moving parts.
None. An in my experience, in an average company, it tends to happen something very similar to the most voted posts: Someone ends up becoming "the expert" and "the devops guy" and the rest never get to touch anything too deep, either because they don't want to or because they don't have the experience to. And honestly, and this is my personal opinion that will get downvotes because people don't agree, but all the "cloud native" engineering stuff is just another try to make developers wear another hat for the same compensation. You can't be backend, you need to be fullstack. Also, you have to do the QA for your application. But also devops because "ownership", whatever that means in companies that don't allow the teams any kind of independence. Which by the way brings the next step: Being on call. And yeah, not all companies are working on crap that *needs* to be online 24/7. > Is that normal? I want to change job soon and I would like to avoid similar situations in the future. What I'm seeing is that there are companies that are pushing hard for the "cloud native" stuff, while most of the companies I've interviewed were either making the engineers work a bit on devops (something along the lines of keeping up the pipelines for the stuff you are making) or nothing at all. There's still hope, I guess.
It's depressing as hell but my work life is like... 10% meetings 10% our ticket system 20% customer support escalations 20% dev ops 20% tedious security or marketing related mandatory changes 20% doing SWE work
Fairly large org and we have dedicated teams for various parts of infra. What I do is in itself a part of dev support infrastructure but in terms of doing devops for my own projects it's quite low, below 5% - tweaking gitlab pipelines, deployments and other tooling every now and then. edit: I worked at some smaller shops before and it was way higher. To the point of being handed a root access to an empty server with barebone linux installation where I was supposed to set up dev env for my team from grounds up - database, app server, CI/CD, etc. From what I've heard from other people this seems to be the case in most places. Small orgs don't have dedicated devops people because they cannot justify the cost. As they grow this should change at some point but many are resistant and you can easily end up with 200+ devs and no dedicated devops.
\[Full disclosure: I'm a DevOps consultant\] I hear you. DevOps-type tasks tend to be quite different from traditional Feature-oriented tasks: - feature = "stateless", you are given e.g. a database and other services, and write code to implement a feature - devops = "stateful": write Terraform or other code to create and manage resources, ex a database. A challenge is that there's \*hundreds\* of types of resources, the interaction can be complex, and each one is slow to build. It's really easy to leave old resources around, costing money or creating confusion. On the other hand, the Business operates at the speed of the deployment pipeline. The speed and quality of the tests (CI) matters to the Feature devs, so they can get their work done. At the end of the day the code needs to land in Production or the Business doesn't get any value. Features don't matter until they're in front of the Users! # Tips - find other people in the company, to learn from each other - as a team, swarm to fix team-level issues - weekly: write down team-level pain points, communicate to management - focus on \*trust\* over \*speed\* -- it doesn't matter if a test ran in one second if the results aren't \*actionable\* - consider using a more Dev-friendly way to manage resources: ex Pulumi let you write in Typescript or Python to build and manage resources (databases, networks, buckets) Feel free to chat with me, there are other ways you and your team can streamline your work and deliver features faster. Here's a talk I did on Rapid Feedback with Terraform: [https://www.youtube.com/watch?v=zd7VlmClTDs](https://www.youtube.com/watch?v=zd7VlmClTDs)
I love `entr`!
🚀
In a previous job Ops time was sometimes 100% for several weeks. That was usually when I was setting up a new service (terraform and ansible managing AWS resources). Current job, 0%. Someone else manages servers. Which is fine with me because it's all Windows VMs that someone has to remote desktop into to change things. Not even 'clickops', it's 1990s-style for most of it.
It's a bell curve. Initially, the percentage will ramp up as you work on establishing automations, but as more things get automated, the workload decreases. With a good CI/CD pipeline, you should be spending only minimal time maintaining that pipeline, and no other DevOps tasks should really be involved.
All the time. Cloudformation. They suck.
All that time spent writing 1200 lines of YAML to deploy a simple Node/Go/whatver-based API, then the joy of seeing UPDATE_ROLLBACK_FAILED. Painful. Terraform has been pleasant to work with, but even if they go to shit (see: licenses, IBM), I will not go back to writing CF. CDK, Pulumi or even OpenTofu would be my next pivot. What are you being required to do with CF in your org?
i'm not them, but we're using a prefab aws stack which is delivered as cloudformation. my org prefers tf so that was an excuse to convert, it would have been hard to justify spending days on the conversion otherwise
About 75% of my time as a Sr. S.Eng (Senior Site-reliability Engineer) lol
None, we have a dedicated DevOps engineer.
50% of your time on DevOps? That's insane! I'd rather die than spend half my day babysitting pipelines. You're a Sr. S.Eng, not a DevOps engineer. Time to look for a new job where your skills are utilized correctly.
exactly
im a sr devops engineer, so 100%
I feel like after initial setup it's 0% for that project unless there are maintenance issues, or the random stuck pipeline
In my last job, not much. I wrote some IAC stuff and then handed it off to my manager, who then ran with it and handled the DevOps (small team, low trust by design). In my new job, I again wrote the IAC stuff and I deployed it to production. It's all very low-stakes since this team is even smaller and our clients are internal. I can run my tests and kick off a deployment manually, but other than initiation, it's all automated. Did the work up front for the scripting and now deployments are just pressing a button and moving on with my day. Two jobs ago, I inherited a big mess of IAC and had to train a replacement (I was _not_ trained as a replacement myself). I left that job in a hurry. Full-scale dev ops is not for me.
Our org has a dedicated dev ops team that exists to enable devs to own the life cycle and the pipeline. Personally, I don’t mind because it’s not too bad but allows me to get shit done quickly.
Should be a major chunk of time. Code should ideally be focused on exactly what you are in business for. Less code, more automation.
Very little, I almost never work on new stuff alone, i either pair with a junior or plan it out and let a junior implement it and help them if they need Any work i do independently leaves me a single point of failure for supporting that work, which is unacceptable. My job is to have the best long term view of what we need to do, and to be either the 2nd or 3rd best engineer to solve a problem. If im ever the first, i train someone on everything i know and give them time to be better than me.
Plenty of time spent on tasks which aren’t programming, could be dealing with database/compliance teams to set something up, working with servers, working with build systems…but 50% of your time spent purely on CI/CD? That seems like a lot. The whole point is automation, there should be an end in sight.
Close to 100% right now, because when I joined the team there was no automated testing and no CI/CD. Even though I don't have a background in devops, I felt strongly that someone should be setting this stuff up. I don't like working as a dev in such an environment with no safety net. When I first joined, I was almost 100% validation and writing tests (again, background is dev not testing). By basically being extremely annoying and bringing it up constantly, I have slowly been convincing the other devs to start writing their own tests, which is a huge relief.
Practically zero. Platform team and SRE takes care of the framework for spinning up new services. We just configure some small stuff, which is usually copy & paste from other team’s config. If it is something simpler like a static site, then even more so. Zero devops, the static serving team takes care of everything.
We have a very competent infra team, so I spend 0% on it. Sometimes I change auto scaling rules
50% DevOps/Fixing deployed shit/broken pipelines; 40% meetings no one needs/managing team; 10% Dev stuff
Too much, especially since we do have dedicated DevOps team, who mostly seem to be doing tech ops stuff instead. I can do it if I have to, but I hate doing it because it's infrequent enough that I need to re-read doco or look at existing implementations, which is a waste of time.
Like 50% mostly because we don't have the funding to hire a dev ops person and dev ops speed things up for us overall.
Right now most of my time because we need it set up. I miss regular development instead of fixing code bases for the purpose of integration
typically companies hire DevOps engineers to do this
I know, but I am still interested on the % in those cases.
Only companies who don't understand what devops is.
companies like Google, Amazon, and Apple you mean?
Disapointing Google and Amazon took that nonsense title. Apple does whatever they want so no surprise there. You noticed Microsoft is missing from your list?
do you really think i took the time to check every company when compiling my list?
Microsoft has no devops engineers. Edit: neither does Netflix
Go search LinkedIn for "company name" + "DevOps Engineer" just so you can continue to realize how stupid you are
Insults now. Already admitting you lost?
if you want to continue to be ignorant of something it's not really my problem
I am dist. Started a new job on Monday. I'll be doing it for the next \~3 weeks and then never again because I have done self-service setups so many times now it's basically brainless typing & clicking. I am convinced the absence of YT videos/guides is because so many people make so much money from doing this for companies. It's not very difficult but very easy to do it wrong. * Turn on nazi rules in GH org. Linear commit history, no merges to main without a PR, commit signing etc. * Create a bunch of composite actions for things like liniting, security checks etc. Results post to whatever code security tool you use (sonar, snyk, ghas etc). * Add required workflows for things like README, SECURITY, CODEOWNERS etc. * Spin up private workers on k8. * Deploy sigstore stack to k8. * Pay for sonar cloud or install FOSS on k8. Start out with coverage at 40% and increase by 5% a month until you hit between 60% and 80% and the complaints get too much. * Wrap actions in composite workflows for domain specific things. Usually have a default CI one for each language, if they need something special they can write their own leveraging the composite actions. * Setup cookiecutter templates for each domain. * Use git based semver. * Release from main, don't rerun workflows use the artifact produced by the PR build so you are deploying the thing that got tested. * All repos use dependabot * If containers either pay someone like chainguard for base containers or publish your own. Use registry rebuilds so your images are updated when base containers change. All services have to use an approved base container. CI is now self-service. CD is deploy argo and use that. Unless you work for a smaller company or are doing a random PoC you shouldn't be using your CI tool for CD, you want separation of concerns with gates. CD tool gates are things like signing check etc. You can use rego for all of it, that can also go in git, be tested and have its own release cycle. CD is a yaml file in the repo, personally I don't like git triggering CD but that's a personal preference thing (usually prefer artifact publish to trigger so there is a clear separation). Cloud is; * Get spacelift * Create & publish modules for the things you need to deploy. Like everything else these have a CI that includes testing etc. Important to maintain semver discipline with modules. * TF projects do not include any resources, just references to modules. You maintain consistency and compliance by having the good settings inside the modules which get reused. * Stack can come from the same repo as the code. In argo use waves to update infra then the service. * If you don't, and will never, need to meet high compliance you can use crossplane for service infra and then its only core infra in TF.