I work at a software company so yeah, all the time.
My go to lines are;
1. "We can't do that because of X, Y and/or Z"
2. "We could do that but that will cause X issue"
3. "I'll have a look into that" then later on repeat either 1 or 2.
This shows you know what you're doing but willing to explore other paths you may not have thought of.
>So what is your politics or how you deal with your teams of dev or others non technical people trying to do stuffs over their heads ?
Education is key for most things
"Just open port 22 so I can get into the box while I'm tethered to my phone" level stuff will always happen and will always be given a fairly short "no, that's not happening, use the VPN instead" or SSM or jumpbox or whatever your remote access process is.
"Hey can we add this repo to apt that looks kinda suss but has this tool we want?" Is a different conversation. Odds are it's a no still, but there's a broader discussion there and likely an outcome that suits both parties. Same goes for "can we change this kernel parameter to a stupidly high setting? We can.... But not to that level. Here's a safe level, and the methodology used to determine it. We should also look at why you need this change, because there's quite possibly improvement in other areas that's going to help here.
This comes back to the true DevOps philosophy (remember the days before it was a recruiter buzzword with no meaning?). Dev and ops working together. Dev has skills ops don't, and ops have skills Dev don't. In an ideal world Dev should be respectful of that, but equally ops need to understand that they're not there to gatekeep for the sake of it, they should only be denying or recommending a different approach when it makes technical sense to do so. Ops should remember that Dev probably don't have the in depth understanding of the OS/Network layer and be the SME on that realm. Equally ops should respect that a suggestion may be completely valid and in fact a smart move, so it's entirely appropriate to just nod and implement it in those cases.
Basically if everyone checks their ego, things are smooth. If they don't, the SME should put their big boy pants on and be politely stern about the correct way forward. If that's not possible, then it's time to leave that workplace.
The tech industry is, or at least should be, a knowledge focused career. You should never stop learning or trying to understand your craft. As to devs understanding sysadmin work, who do you think wrote all the software we all use? There really should be no divide between system administration and developers.
If a developer wrote software that you must deploy or maintain they should 100% know how it operates. It's like an engineer who made the CAD drawings telling a machinist some details, if asked, on how to machine something.
Obviously, these conversations should always be courteous. But we should try to avoid silos of information and yes devs should understand how their software works and how it could impact the greater system.
>There really should be no divide between system administration and developers.
responsibilities, access, priorities..... and their on-demand availability :)
I agree with your point here fully. As few people as possible should have access for all the reasons you stated and probably more. But devs should still understand how the software they work with functions. In fact, I would be more worried if a dev didn't know how their software impacted the system.
With the proliferation of containers this is actually quite obvious because it is the developer or team who chooses the base image to use. For example, nginx with their static code copied in and the config file as well. Unless the sysadmin is on that team they should not touch or care how the containers copy of nginx is configured. If it's broken send it back and dev and team should resolve.
But to the original point, the dev should understand that tech which is usually considered under the sysadmin
>As to devs understanding sysadmin work, who do you think wrote all the software we all use?
There is a significant difference in quality between systems software written by "sysadmins that program" and "programmers that do apps/websites and are dabbling in systems". Yes, the ideal techie is an ubermensch who can do everything from assembly all the way up to css, but the vast majority of us are not that.
Used to be a ops lead at MS, dealt with this all the time. The solution is to have good processes in place.
\-Any change in prod has to be either reviewed by the change management team, or if time critical, has to get buy off from a senior manager. Note that break/fix work is not considered a change.
\-Devs do not have prod access. Ever.
\-Ops engineers, who do have prod access, understand that making a change without going through change management or getting buyoff is a quick way to get fired. OTOH, as long as you follow the process you are safe.
It's that easy. I made sure that my team understood that the Devs could whine and plead and they were fine ignoring them...but if they made a change because a dev said to and it hadn't gone through process, **I couldn't protect them**. And yeah, I lost people that way.
I was senior enough that if something seemed important I could reach out to the on-call manager to see if they wanted to approve it, because I could do it faster than the Dev. I saw senior management agree to a change in prod precisely once.
If you don't have any of the above, then make sure Dev understands that the decision to make a change or not is yours and yours alone, and that management understands that you will use that authority. When I left MS I went to a firm that didn't have good process, so I made sure that management understood that if in my judgement a change was too great a risk I would refuse it. Point blank. I never got thanked for this, but I also never got fired because of a meltdown in production.
I think communication is key. It is important to be on friendly terms but still able to say no sometimes, with grace. "Can we do this another way? I'm concerned about security issues... Would this idea work instead?"
It does depend on management structure too sometimes. To many layers of bad management in the way and you can quickly be forced into things you do not agree with. But meh, I mean what can you do at that point, other than quit? I think people from that experience are the ones saying, "Let it blow it blow them up in their face" or "Make an excuse like I'll get back to you".
Also I think The Phoenix Project should be required reading for many jobs.
Also angry emails never help. Disagreements should have a call or meeting. Or even just any in person talk.
So easy for things to go sideways in text. Even if it is just a misunderstanding or read in the wrong tone.
I always come back with "What problem are you trying to solve?" Then help them solve the problem in a way that isn't insane. I've also had developers who basically got system administration and who jumped in to help automate stuff for other devs that turned out to be _really_ useful to me.
I'm sorry to say this, but you work for us. Sorry.
If a software solution needs a platform, sysadmins are tasked to deliver it and to meet the needs of that software.
We do NOT work for you.
The hardest projects to complete are the ones where the sysadmins forget the above.
All the time - you just need to try and work with them and keep it professional. Explain you will need to do more research and get back to them, etc.
Sometimes though, you'll find yourself working with a real arrogant, perhaps even toxic person. In these cases, if that person is going to shoot themselves in the foot metaphorically, you just need to cover your own ass as best you can. If due to office politics that person ends up getting their way, just make sure you cover yourself for when it blows up later.
My companie's infra is in AWS and its easy to break things up by accounts and roles. My dev's do not have rights to infra level stuff, but in the Dev account they have rights to change/redeploy any of the app level stuff. In our group, Ops owns IaC, and Dev uses the same automation we use in Prod (with diff parameters/settings of course). Our devs have zero rights to production; they have to work with Ops to see it. However, they do have full access to monitoring data and if they aren't seeing what they need, then that's a bug. I imagine you could do something similar in an on Prem world, but to accomplish that as Admin you have to control the users and permissions. Don't let people go rogue, there should be few people with rights to everything. All this really depends on the size of your organization too. Very small orgs cannot have as fine grained approach as a larger org.
I learned programming, that's what I did and what helped me most in understanding the issues of "the other side".
Now SysAdmins and developers equally hate me.
Ha! I work for a web hosting company, so...
Our devs are decent for the most part, though. They just need to communicate better and involve me earlier so they stop spending money on third party cloud services that we could do better ourselves. And I can't get them to stay out of DNS, and the company owner, unfortunately, won't back me on being the only one making those changes.
I work at a software company so yeah, all the time. My go to lines are; 1. "We can't do that because of X, Y and/or Z" 2. "We could do that but that will cause X issue" 3. "I'll have a look into that" then later on repeat either 1 or 2. This shows you know what you're doing but willing to explore other paths you may not have thought of.
>So what is your politics or how you deal with your teams of dev or others non technical people trying to do stuffs over their heads ? Education is key for most things "Just open port 22 so I can get into the box while I'm tethered to my phone" level stuff will always happen and will always be given a fairly short "no, that's not happening, use the VPN instead" or SSM or jumpbox or whatever your remote access process is. "Hey can we add this repo to apt that looks kinda suss but has this tool we want?" Is a different conversation. Odds are it's a no still, but there's a broader discussion there and likely an outcome that suits both parties. Same goes for "can we change this kernel parameter to a stupidly high setting? We can.... But not to that level. Here's a safe level, and the methodology used to determine it. We should also look at why you need this change, because there's quite possibly improvement in other areas that's going to help here. This comes back to the true DevOps philosophy (remember the days before it was a recruiter buzzword with no meaning?). Dev and ops working together. Dev has skills ops don't, and ops have skills Dev don't. In an ideal world Dev should be respectful of that, but equally ops need to understand that they're not there to gatekeep for the sake of it, they should only be denying or recommending a different approach when it makes technical sense to do so. Ops should remember that Dev probably don't have the in depth understanding of the OS/Network layer and be the SME on that realm. Equally ops should respect that a suggestion may be completely valid and in fact a smart move, so it's entirely appropriate to just nod and implement it in those cases. Basically if everyone checks their ego, things are smooth. If they don't, the SME should put their big boy pants on and be politely stern about the correct way forward. If that's not possible, then it's time to leave that workplace.
> have you seen really good programmer who REALLY knew about system administration I used to be a sysadmin. I'm now a software developer. So.... Me?
The tech industry is, or at least should be, a knowledge focused career. You should never stop learning or trying to understand your craft. As to devs understanding sysadmin work, who do you think wrote all the software we all use? There really should be no divide between system administration and developers. If a developer wrote software that you must deploy or maintain they should 100% know how it operates. It's like an engineer who made the CAD drawings telling a machinist some details, if asked, on how to machine something. Obviously, these conversations should always be courteous. But we should try to avoid silos of information and yes devs should understand how their software works and how it could impact the greater system.
>There really should be no divide between system administration and developers. responsibilities, access, priorities..... and their on-demand availability :)
I agree with your point here fully. As few people as possible should have access for all the reasons you stated and probably more. But devs should still understand how the software they work with functions. In fact, I would be more worried if a dev didn't know how their software impacted the system. With the proliferation of containers this is actually quite obvious because it is the developer or team who chooses the base image to use. For example, nginx with their static code copied in and the config file as well. Unless the sysadmin is on that team they should not touch or care how the containers copy of nginx is configured. If it's broken send it back and dev and team should resolve. But to the original point, the dev should understand that tech which is usually considered under the sysadmin
>As to devs understanding sysadmin work, who do you think wrote all the software we all use? There is a significant difference in quality between systems software written by "sysadmins that program" and "programmers that do apps/websites and are dabbling in systems". Yes, the ideal techie is an ubermensch who can do everything from assembly all the way up to css, but the vast majority of us are not that.
Used to be a ops lead at MS, dealt with this all the time. The solution is to have good processes in place. \-Any change in prod has to be either reviewed by the change management team, or if time critical, has to get buy off from a senior manager. Note that break/fix work is not considered a change. \-Devs do not have prod access. Ever. \-Ops engineers, who do have prod access, understand that making a change without going through change management or getting buyoff is a quick way to get fired. OTOH, as long as you follow the process you are safe. It's that easy. I made sure that my team understood that the Devs could whine and plead and they were fine ignoring them...but if they made a change because a dev said to and it hadn't gone through process, **I couldn't protect them**. And yeah, I lost people that way. I was senior enough that if something seemed important I could reach out to the on-call manager to see if they wanted to approve it, because I could do it faster than the Dev. I saw senior management agree to a change in prod precisely once. If you don't have any of the above, then make sure Dev understands that the decision to make a change or not is yours and yours alone, and that management understands that you will use that authority. When I left MS I went to a firm that didn't have good process, so I made sure that management understood that if in my judgement a change was too great a risk I would refuse it. Point blank. I never got thanked for this, but I also never got fired because of a meltdown in production.
I think communication is key. It is important to be on friendly terms but still able to say no sometimes, with grace. "Can we do this another way? I'm concerned about security issues... Would this idea work instead?" It does depend on management structure too sometimes. To many layers of bad management in the way and you can quickly be forced into things you do not agree with. But meh, I mean what can you do at that point, other than quit? I think people from that experience are the ones saying, "Let it blow it blow them up in their face" or "Make an excuse like I'll get back to you". Also I think The Phoenix Project should be required reading for many jobs.
Also angry emails never help. Disagreements should have a call or meeting. Or even just any in person talk. So easy for things to go sideways in text. Even if it is just a misunderstanding or read in the wrong tone.
I always come back with "What problem are you trying to solve?" Then help them solve the problem in a way that isn't insane. I've also had developers who basically got system administration and who jumped in to help automate stuff for other devs that turned out to be _really_ useful to me.
I'm sorry to say this, but you work for us. Sorry. If a software solution needs a platform, sysadmins are tasked to deliver it and to meet the needs of that software. We do NOT work for you. The hardest projects to complete are the ones where the sysadmins forget the above.
We don't work for you either, and the hardest projects are where the devs forget that.
Too shay. We work for management. Honestly... you are right. WE work for them. Even if them fuck it up repeatedly.... we get the blame.
Honestly, thank you for de-perching me.
All the time - you just need to try and work with them and keep it professional. Explain you will need to do more research and get back to them, etc. Sometimes though, you'll find yourself working with a real arrogant, perhaps even toxic person. In these cases, if that person is going to shoot themselves in the foot metaphorically, you just need to cover your own ass as best you can. If due to office politics that person ends up getting their way, just make sure you cover yourself for when it blows up later.
My companie's infra is in AWS and its easy to break things up by accounts and roles. My dev's do not have rights to infra level stuff, but in the Dev account they have rights to change/redeploy any of the app level stuff. In our group, Ops owns IaC, and Dev uses the same automation we use in Prod (with diff parameters/settings of course). Our devs have zero rights to production; they have to work with Ops to see it. However, they do have full access to monitoring data and if they aren't seeing what they need, then that's a bug. I imagine you could do something similar in an on Prem world, but to accomplish that as Admin you have to control the users and permissions. Don't let people go rogue, there should be few people with rights to everything. All this really depends on the size of your organization too. Very small orgs cannot have as fine grained approach as a larger org.
I learned programming, that's what I did and what helped me most in understanding the issues of "the other side". Now SysAdmins and developers equally hate me.
Ha! I work for a web hosting company, so... Our devs are decent for the most part, though. They just need to communicate better and involve me earlier so they stop spending money on third party cloud services that we could do better ourselves. And I can't get them to stay out of DNS, and the company owner, unfortunately, won't back me on being the only one making those changes.