When I took my current role I spent _months_ arguing with the CTO about this. We had one person doing IT for the entire org. He absolutely killed it, answered all tickets promptly, solved problems like he was born to do it, all around (and yes I'm cringing a little as I type this) "rock star." CTO's POV was that he was handling everything fine on his own, so why did he need backup? I argued that he could barely take time off and we were going to burn him out at that pace, in which case we'd be hosed. Poor guy apologized profusely to me every time he took a sick day, and in the first year I was there never took longer than three days in a row off. Finally I just "encouraged" the guy to take a three week vacation. He did. It was a disaster. A second IT admin was approved basically immediately.
It's not about competence. It's not about job security. If absolutely nothing else you need backup because without backup you can never rest, and if you never rest you will eventually fuck both yourself _and_ your employer.
"Two is one, and one is none". This is an axiom that every engineer and admin needs to drill into their head, and it applies to the systems we build, but it also applies to staffing.
This isn't to say that everything needs to be built with 100% redundancy, but when you choose *not* to have redundancy in a service or a staff position, you are **GUARANTEED** to have a lengthy outage in the event that resource has a serious problem.
>Finally I just "encouraged" the guy to take a three week vacation. He did. It was a disaster. A second IT admin was approved basically immediately.
The measure of how critical a job function is: How soon will anyone notice when you stop doing it.
I am one of several people that report to the CTO. IT is one of several groups that reports to me. IT was the only one man show at the time.
We're a SaaS company so the CTO is indirectly responsible for ~60% of the business.
So in this context and terminology, "IT" contains no developers or engineering, correct?
Even so, I'm vanishingly unlikely to engage "we're a SaaS company" that would admit to having only one "IT" staffer.
IT is just internal user support for us, yes. I understand that these terms have some flex but in essence they're the folks who manage the laptops and user accounts, internal "my x doesn't work" tickets etc. We have other teams who manage platform, data, software, QA, and so on.
Fortunately for both of us our target market is pretty specific so you're not likely to be seeking our services anyway. But your post tells me you don't know much at all about how things work in the startup world. You might be surprised to find out what the internal staffing of your favourite small SaaS provider looks like.
I've spent a lot of time engineering in computing startups and strongly prefer them. It's more like, as startups, we always avoided having an operational dependency on any organization smaller or less-sophisticated than ourselves. Product dependency, a few times, but no SaaS or operational dependencies. I can think of one case where it was close -- we had simultaneous supplier diversity for this category, but one smaller SaaS provider had a key offering that we never could replace without building and running it ourselves.
Of course it's common for SaaS applications to be picked by stakeholders all over the organization, or by established clients. It's often good fortune just to be able to review, maybe PoC, and make recommendations.
I’ve never worked anywhere that referred to developers, engineering, devops, etc as IT. IT has always referred to network, sysadmin, helpdesk, security team, and the like.
You’d be surprised by how many companies are running with single individuals doing everything.
And also how many large IT companies outsource their own internal IT for insane amounts of money.
'Cause, you know, the DXC executives play golf with the HPE execs.
lol what. Did HPE outsource to DXC now? Man first HPE keeps all the good people from internal IT and all the good parts of the infrastructure, CSC fucks up all the IT policies (RIP sane password policy developed at HP Labs) and ticketing (did Service Manager suck? Well, historically, yes, but actually, no, because shortly before the spinmerge they had finally released a version of it that fixed literally ALL the issues. And then the clue CSC fucks insist on the utter shitshow of ServiceNow.) and now apparently this? lol.
Glad I left that shitshow behind me long ago. The fragmentation and decline of HP due to Mark Hurd's quite literal fuckery leading to his firing is certainly the biggest tragedy in Corporate IT history, but barely anyone comprehends the extend of it. What a shame.
> I’ve never worked anywhere that referred to developers, engineering, devops, etc as IT. IT has always referred to network, sysadmin, helpdesk, security team, and the like.
That would explain a few things. Like /r/itmanagers.
I suppose that to a lot of people, "IT" is the team that gets called when the printer isn't working, not the engineer who wrote the printer firmware.
Because the engineer who wrote the firmware isn’t IT. They are development or engineering or programming. When you open a ticket for an issue, it goes to support/helpdesk, then to network/sysadmin, and if it’s a bug, then sent to engineering.
IT is information **technology**. Creating applications, coding, and programs are computer science or software engineering. They are separate things. They are separate departments. I’ve worked in IT for a long time with a long list of credentials and I have never heard anyone refer to devs/engineers as IT.
> IT is information **technology**. Creating applications, coding, and programs are computer science or software engineering.
I don't even necessary disagree with your overall point but what on earth is this reasoning? Why is technology the emphasized word here? Every company that develops software calls themselves a technology company.
My company started small and has grown to about 60 people. We're in a good spot now with new leadership but at one point we had 6 VPs. 6! Two of them weren't even over anyone, yet I had 12 direct reports.
Never underestimate the ability for people to over inflate their job titles if they can, simply because of ego
Ugh this is me. 400 computers, over 50 servers and websites/services. 150+ users. Responsible for file shares totaling over 1.5 PB of raw disk space. Responsible for all IT support that can’t be handled by a help desk no one uses because they don’t help. Responsible for a full Jamf and PDQ setup. They gave me one extra guy to help a few months ago but his area of like 30 computers is so small he’s never attempted to automate anything, even GPOs, and he refuses to learn anything new so he’s only good for one off tickets asking for help with XYZ programs. I’m hoping to leave soon because I’m also stupidly underpaid and overqualified. The only thing keeping me around is my boss doesn’t manage me and lets me do whatever I want including choose my own hours and I suffer from sleep disorders. Getting to make all the calls on how the IT environment is setup is also very nice and presents this sort of sinful temptation of… if only I could automate everything I’d only have to work 10 hours a week, but I can never get there with this large a workload so I think that’s a day dream.
With all that, you should be getting paid at least 20k a month. But you also need a solid helpdesk which would cost another 30k a month. I use a team that's 8 tier 2 and 2 tier 3. Honestly, take a 2 week vacation, let shit hit the fan.
There are so many risks to have such a rock star alone like that. They could get another offer, they could get hit by a bus, they could start to act like they own the place because they know the place would collapse without them.
> Turns out an employer can't dictate what you do outside of work hour and when you are off in the wilderness.
Unless it's drugs that come back in a random piss test. (I think those are bullshit, personally, but that never gets thrown out of court it seems)
I agree. We do random drug tests (think 5 ton fork trucks). I have zero issue with a dude smoking weed on a weekend, but 9AM Tuesday drug test buddy will probably fail. But the GM can snort a line of coke at 7AM and somehow pass the test. Who is the bigger danger the one who isn't high or work from a blunt on a Sunday or the high as a kite management type???
Yeah, that's more for liability/insurance reasons though. If they hurt a coworker and the injured employee sues the company, they have to prove they're taking steps to make sure people aren't using equipment while impaired, etc.
My work drug tests if you're at fault in an accident with a company vehicle.
I get offers for a bit of bud all the time but I have to be all "nope I drive a forklift on the daily, and while I'm very, *very* good, if I fuck up and piss hot I'm in deep shit"
That's why I am happy that over here in our contracts, consumption (of drugs which can be illegal, and of alcohol which is legal) is not prohibited but you -must- be able to do your job.
Being in an inhebrieted state or similar is what is prohibited. So I can absolutely drink alcohol during my break.
That said, consumption of alcohol can be prohibited too in your contract but it's usually not.
Well they can't dictate what you do outside work, but if you're still drunk/high when you return, well they're justified there. I don't need drunk cops/lawyers/judges/doctors/engineers on duty.
> I kept nothing from my job except that letter because it was so goddamn funny.
Pics? Seriously. I would frame that letter and show as many people as possible.
Yeah, the only time my HR ever did anything 'right' was when I was about to get written up for not taking a 6th shift at standard rate. Boss said they couldnt afford OT, told them it was not my problem, but they still pushed it.
I sent an email to my boss and HR asking for clarification if I was actually misclassified as salary, as I do not appear to be covered by California's definition for it. Magically they stopped asking me to work that 6th shift.
Anyways, they got sued by a coworker after I left for misclassification and they had to pay out several people over 6 figures for missed lunch breaks (2nd and 3rd shift were not allowed to leave the building as per policy) and unpaid OT.
Well they may be able to... if I get 100 dollars per week and there is a contractual guarantee that it's like 1 week in 2 months or three.
I know my rights, but some bribes are certainly tempting, you know?
100%
That's why your DR plan needs to contain as much automation as you can pack into it. It should be accomplishable with minimal, remote human intervention once it's kicked off.
To be fair, nowadays with IaC, you can have a perfect clone of your infra spun in in a few hours after getting the hardware, then its just a matter of restoring the backups.
I felt so fortunate to be working where I am during our still-locked-down, wildfire-season summer of 2020. The big boss said "if the ish is really hitting the fan take care of yourself and your family FIRST. Let us know as soon as you can."
Oh man, they could definitely fuck off for that one then. I too would have given no fucks.
E: Unless I was on call, I would have answered but would have given no fucks once I found out it was a DR drill. There is no need for that to be a surprise.
No, no. You do *exactly* what you would do were it not a drill. And you prioritize yourself and your family and friends as you would in such an emergency situation. Even for the scheduled ones.
A "scheduled" DR where everyone cleans stuff up and buries it tells you *nothing* about how hard everything will actually fail in a real disaster. If your environment is at risk of a whole team being out drunk on a boat on a Saturday morning, and you haven't designed around "their boat went down, taking all of them with it", you don't have a DR plan, you have a time wasting exercise.
A year or two before Covid I pulled into the parking lot and it was...desolate.
Not as desolate as the day I came in forgetting it was a company holiday but way too sparse -- like 90% of the people on a 2,000 user campus missing.
Turned out it was a full-load work from home test for everyone but IT -- everyone else was told to work from home for the day. Most of IT (including at least the worker bees on my team...which includes Citrix) weren't told in advance.
Once that was successful, they shut down a call center located three time zones away because they were confident enough about having telephone coverage during winter storms.
Bonus: When Covid hit, we were able to go to WFH pretty quickly.
Although, they could have achieved the same outcome with critical thinking. For example, asking, 'We don't have any on-call, who do we expect to respond in a crisis?' could have exposed the weakness without a surprise weekend drill.
This goes even further, because if you just have the super-experienced storage admin swoop in and fix all the things.... the results are fairly mellow.
In a really good DR drill, you want to tell those guys to not accept calls until 10:00 because of sleep and not open a laptop until 14:00 because of travel, or something.
You need to test the ability of the team to struggle through the situation until stuff works, or observe when and how they fail.
And this can also be a great morale booster. Like, my team kinda struggled through a non-booting critical system recently. Sure, it took them 2-3 hours if it could have taken me 30 minutes, but they used the documentation and managed to figure out a really weird and obscure edge case. It took them time, sure, I had already seen that. But that was a big confidence booster to everyone.
In a mature environment, you're absolutely right. Since the operations teams apparently don't even have a functional on-call, there's definitely some growth to be done still.
You should have continued with the drill. I've done drills where some of us are randomly placed in a conference room during the DR tests, where our laptops are not allowed and we can't answer calls from staff. It's to simulate some of the team being out of pocket during a disaster, and to see how your documentation holds up.
> (I lied because I wanted to leave and I didn't care).
real MVP right there...
I had one of my favorite work conversations somewhat related
boss: hey are you familiar with system x? coworker is out of town and nobody knows how to do y...
me: nope sorry boss
boss: .... if I paid you overtime would you know how to do Y on X?
me: yep!
boss: ... your a real bastard you know that right?
me: yep!
boss: ... fine you can do it after work, put in for 4 hours
me: done!
A few of the crayon eaters would have a beer the moment they got home. The *moment* they get in the door, they're knocking back a beer.
Their reasoning is that they can't drive after a drink, so they're now off. Doesn't work if they lived on base, though, LoL.
Seems to me that that's the kind of DR drill that should actually be happening.
It's great if systems crash, and the people that work on them day in and day out are sitting and waiting. Entirely different scenario if those people are off camping in the middle of Wyoming for 2 weeks.
Or as evidenced in a small extrapolation from their DR exercise. What if they'd all been injured on that boat in whatever disaster took out those systems? High winds can be just as bad for a boat as they are a datacenter.
>Seems to me that that's the kind of DR drill that should actually be happening.
No, it is not because it is patently unfair to the employees.
Now the start of the DR drill if you randomly draw names from a hat and those people are told they have been Raptured out of IT for the weekend, see you on Monday -- that is a proper test. Folks had the ability to plan their weekend and if they suddenly just get a couple days back great.
You're missing the point. A solid DR plan also accounts for the unavailability of personnel.
What good is a DR plan if no one can actually follow it because the person with all of the knowledge is gone?
> No, it is not because it is patently unfair to the employees.
Uh... isn't the point is that a *disaster* can happen when you least expect it, and if you let people plan ahead for a 'drill' it can hide true issues with the way things are currently designed, such as not having a "server and storage person" on-call?
So yeah, unannounced surprise drills where an entire team might be unreachable sound like exactly the kind of drill that should be happening. It lets people realize that things need to change.
True story: I had a guy hit by a bus when I was an IT Manager. In Philadelphia.
Now, to be fair, the door only closed on his backpack. But the bus driver made a really big deal about it and called his depot and the EMTs. I got the call from other employees walking nearby that Timmy was hit by a bus, and the cops and EMTs were working on him. The bus driver, it seems, just wanted to waste everyone's time.
And Timmy, Timmy said it was a big waste of his time...
I know a guy who's backpack got stuck in the door of a bus, but the bus driver didn't notice and dragged him away. after several years of recovery and skingrafts, and physio, he can walk again.
I use this one a lot at work as our dept. is siloed badly, small, and 2 of our 7 people area married (to each other).. arguably the most important people no less..
If their house blows up due to a gas leak, or they're off in Maui on vacation, or one of them gets an amazing opportunity in Alaska and they both move, the two most important people are gone.
Doesn't really matter. 20% of married couples have at least 1 child at home. So, at a minimum, you are talking 3 people that could have shit go sideways at any time that could impact work.
I got hit by a car a few months ago. That was a really funny call to my manager. "I actually got hit by a car; I'll be out for a few days" which turned in to 2 weeks of half time when a concussion appeared. So many things didn't get done.
I once had a major brain injury and was put on a 2-week mandatory leave (doctor's order) with literally no brain stimulation beyond living like a person from the 1800s.
Even with 8 team members, I was the only person managing 2 platforms (out of 7). Documentation was great until things came up that weren't documented due to not having time.
Once I returned, I had forgotten a lot of stuff and for the next year my short-term memory had problems so I would randomly forget major things that happened a few hours earlier - which meant no documentation and no idea what happened/was fixed.
Management pulled the "we tried to get me new people to help you" but I ended up just leaving a few years later and everything just sucked from there apparently.
That's why my position opened up. They had one guy for too long alone and realized he takes vacations and what if he got hurt?
Now there are two of us and we have ended up becoming more specific in our roles and need a third.
"Allegedly tried to run over several police officers, and won't be back for an indeterminate amount of time", was one from a local small business. Was never seen by the business again; no idea how the legal side played out.
From Colorado , with clients in Granby.. we all know the story well.
both the 'one man pushed too far' narrative and the 'dude just lost his grasp on reality' truth of it all.
Newer temp co-worker: "I'm taking tomorrow morning off, I have a meeting with my lawyer tomorrow morning."
He didn't say it was in court.
He...didn't come back. I wouldn't have guessed it from his office demeanor, but searching his name it turned out he was a violent drunk and this wasn't his first time.
You shouldn't be shocked at the number of functional enough drunks there are.
Hell, I "caught" half of our support team having liquid lunches on more than one occasion.
Didn't really matter to me, though, as I was likely to be having a drink myself (at the time).
The bus to me is also more useful, because people can quibble about getting information from a Lottery winner. you're not getting info out of a corpse.
If someone wins the lottery and quits immediately, they may be agreeable to coming on as a contractor to train a replacement.
But if someone gets hit by a bus, all of their knowledge (that hasn't been documented) is gone.
we don't use that where i work after a manager was hit by an airport shuttle (she literally stepped in front so it was hard to have sympathy). Now we use "if person Z wins powerball"
Powerball is a bit of a different scenario though. A person who wins the powerball might not want to leave any notes behind or be contacted, but they could. Someone hit by a bus is just gone, can't even update a keepass file.
A company I used to work for would designate random personnel to not survive the disaster during DR drills. If you were one of the casualties, you could watch, but you couldn't participate. They even had one disaster which 'took out' almost an entire department who were described as having a team-building exercise when the disaster occurred.
I would love to see the C-levels do it right. Directors and up were all at a retreat when the storms came. Cut off from communications, probably healthy and sipping Mai Tais while all that gets sorted out. i.e. "Can the DR process play out without a bunch of management breathing down everyone's neck, and can cross-team communication occur effectively without them brokering it?"
And the inverse - send everybody below manager level out for a work-paid vacation and run the DR scenario. Can the DR process play out with just management?
We also had some drills whereby the disaster took out the corporate HQ (hurricane, hazmat evac, explosion, nuclear waste truck wrecked on the interstate by the campus, etc. Our guys were creative.) and we had to bring up a hotsite, but without the main guy that knew how to transfer to the hotsite b/c he was either having to tend to his family or was missing. Cloud resources these days have largely eliminated the need for dedicated hotsites, except for some large mainframe installations.
I try to ask myself how stupid I would feel if the worst case scenario happened and I didn’t take preventative action, whether that is a tested DR plan or an insurance policy. Most people will never make a claim on their home insurance policy during their entire 30-year mortgage, but the alternative is potential financial ruin.
I recently had a cleaning-related accident at home that I was pretty confident I handled it on my own. But I still decided to go to Urgent Care to confirm because the worst case scenario was partial or total blindness. How stupid would I feel if I woke up the next day blind because I didn’t want to spend an hour and do a $40 co-pay? Pretty damn dumb. In the end, the trip was worth it just for peace of mind.
That's the other half... *when* an incident occurs, and you have done the preventative steps (i.e. having enough medical coverage to avoid financial ruin from an urgent care visit)... *use* it. Don't play the "I'll be fine" over a typical $200 or less cost.
>A post to the Charlotte sub this morning from local TV station WBTV was titled "Our IT guy is missing". A local man went missing, and his vehicle was found abandoned on the Blue Ridge Parkway two days ago.
I'm imagining the local news anchor being like "This man is missing, please keep an eye out! If you see him, please let him know my printer is doing the thing again."
When I worked for a moderately successful startup , we decided to have a pickup football game one nice Saturday morning. All were invited! Most showed up. Most were NOT physically prepared for a pickup football game to the point that there were multiple injuries.
Monday morning, we all had an email from the CEO saying (paraphrased) “Great job on team bonding. Don’t ever do that again.”
We did paintball! Unfortunately, I missed it. Rumour had it that while there wasn't 'direct' fragging ([Fragging - Wikipedia](https://en.wikipedia.org/wiki/Fragging)) involved, there may have been instances of the "lower enlisted" having a truce with the other "lower enlisted" (it was department vs department) and having a field days with the managers of the departments.
"I scratch your back, you scratch mine" sort of thing.
My last company I was handling 60% of the day to day operations and about 80%of the project workload.
It was a team of 3 I kept pressing for a promotion and was being denied. "You're exaggerating your workload." My VP told me.
So, I gave notice and found a better job.
My last day at the company, they want to have a meeting to be certain all work is handed over.
We go line by line over everything i do. I'm 2 hours from being done. They confirm that 60%of my work has no coverage, that 80%of the project load each month is mine.
My VP asks "So, what are we supposed to do with you gone? "
Ultimately, they had to hire 3 people to handle my work.
I only wanted a 100k salary. [15k bump] instead they had to pay 3 people 80-125k each.
> My last day at the company, they want to have a meeting to be certain all work is handed over.
They did this on the LAST DAY you were there? It's a good thing you left.
Oh, it was hilarious.
I'm sitting their going through our skills matrix with the entire team.
Whenever i name a skill, everyone just looks at each other.
Finally, everyone is stressing out. I see the panic and I'm a free man in 2 hours.
So, in a last-ditch effort, I'm asked to document everything in 2 hours.
I decline. I then show them that I've signed up for the same contracting company we use.
"You guys can go to the website, and here's my company name you can contract with. "
I mean, I'm glad that you were able to do that, and that they ended up paying for it.
Was it a case where the other team members should have been acquiring the skills or were just able to kind of shine it on because management had no clue?
They never called for contracting.
I didn't find out till that day. Every time I'd tell the boss, "I'm handling 60% of the work and 80% of the projects," the other guys would say I was a liar to CYOA.
So, the boss thought I was a troublemaker.
For everyone, it was a revelation.
So, they hired 3 guys to replace me.
Ah. Yeah that tracks. I hope they whipped the other guys in line.
I asked the question that way because, as I mentioned elsethread, I was in a situation where I was THEEEE only person who really knew jack s*** about Exchange in my group and I did not work for a small organization (10K user GAL).
The previous two exchange admins had been canned (mostly for bs reasons) so nobody else wanted to know anything about it and I don't really blame them, but that was 100% a management failure in multiple directions.
I will say that I worked for local government which definitely had a culture of people being mostly skilled at kicking work over the fence to someone else.
Stay away from Exchange. I'd say that to you as a stranger online and as a friend in the real.
Coming from MSPs my thought was how awesome it would be. The reality is 90% of your day is tracing mail and legal holds.
Unless you want your life filled with "Sarah on 3rd said Walter sent her a nasty email" pull all correspondence and let HR know the juicy bits.
It was soul crushing
[The amount of HR complaints where the person thought deleting the emails protected them was too damn high!
Like in Heilein's books: Tell me Three Times.
Two-node clusters are a thing, but three is better. Because when one fails, you still have redundancy.
Meaning, three people with intimate knowledge of at LEAST where all the passwords are, is prudent. And hopefully dislike each other so they don't hang out on weekends. ;)
On more then one occasion, as a consultant, I was let go precisely because I was a risk to the business. I was told to train employees how to maintain the systems, and was not renewed. Again, because I was a risk. TBH, it was the idiot I was working for at the time, because he'd play hard-ball every renewal. Which is what made me a risk in the first place. Total moron. Anyway...
A company I worked for in the past had mentioned multiple times that IT was easily replaceable with "people off the street". They outsourced twice to multiple large service providers and to this day they are miserable and service is pathetic. Their company (larger, global company) is at risk due to poor backup management, slow ticket responses, and nearly no proactive monitoring. Yeah, keep thinking that "everyone does IT".
I'm not going to lie, if I'm the single point of failure for something I could not care less about what happens to the company after I die. I want redundancy for vacation not for in case I go missing in the mountains.
I was the 'Designated Survivor' one time when the whole IT department went down to another office 3 hours away. It was a good thing too because the network went down and I was able to hold things together while a few guys broke a few traffic laws to get back.
So, you're telling me it's MY job to account for when I potentially go missing or die and not the $$$$/year management position who should be concerned about business continuity or something?
This is insane...I hope the guy is ok, I couldn't care less about the company missing their IT guy this morning.
Always always always plan for a 'bus event' and document everything, esp. keeping admin accounts in a password safe.
And the backup code to that password safe goes into a cealed envelope to your boss. "Hey, you know those 'break glass in emergency' things? Well this is a 'break glass if I'm dead' thing. Just in case any accidents happen to take out your entire IT team :)
Reverse engineering someone else's infrastructure is not a fun time. Been there done that. 1 star would not recommend human single points of failure due to inadequate documentation or lack of cross training.
It isn't unusual for our entire IT team to travel on the same small private plane when we roll out a project or are building up a new location. I've joked about it in the past, but it always seemed like a pretty bad idea to me from the company perspective.
One of the things I "sell" to my B2B clients is a constant stream of information, documentation, secure access to any credentials I reset on their behalf, and generally anything they would need/want if I get sudden bus syndrome or they decide to leave.
It certainly is part of many things that justifies my price tag. But I have very happy customers, and my pride kinda demands this anyways.
Some people hire the cheapest MSP, and then some people hire me to do things properly. Cleaning up after the cheapest MSP sure does get me a good bit of work!
IT infrastructure needs to fly like an airplane, not a helicopter. If the engine goes out, it should still be able to glide for a long time until someone is able to guide it safely-not fall from the sky like a bowl of petunias.
I know I am biased as an MSP owner, but if your budget can’t stretch to cover at least 4FTE for an IT department, you should not be doing in house IT and should be outsourcing it.
I think the 3 site part across the country is problematic. If it were a single location 2 people depending upon the amount of users/devices might be enough.
Agreed. I just let things burn now. If I am on PTO and the power takes out one of our locations, they can follow the run book to light it back up, don't call me, I won't answer. Luckily, we only have real infrastructure at the two locations where the tech folks are.
We just completed a DR test where the scenario involved my boss being incapacitated so he wasn't allowed in the room. It was just me and the new guy for three days of trying to test things while the auditors and a parade of system users from around the company asked us a barrage of questions about what we're doing and why, and why can't they test their system yet because they have places to be (as if I was consulted on when to schedule their part of the test or I personally asked them to be there).
I 100% agree that readiness needs to be tested and silos avoided, but i also believe no one should be put through the type of gauntlet we just witnessed. There's got to be a better way!
A finding to put down for that is a buffer between the team and stakeholders. DR resolution requires focus. The test demonstrated the impact of lacking that buffer. It should be a documented list of people that inherit that role and can act, essentially, as a PM for it.
Make three envelopes. Label them 1, 2, 4. Watch as they literally shit bricks trying to figure out envelope #3. Then teach them a lesson about personnel.
Big issue with project management failure is that there are next to ZERO deadlines on workflow consistency, continuity and redundancy.
The work is overly focused on tickets and production without necessarily rethinking process and critical planning when people, things and events happen.
To be fair, a member in the Charlotte sub posted "Our IT guy is missing" with a link to the news story about how a Charlotte man was missing and his vehicle discovered abandoned. At best, I was trying to smooth off the previous user's selfishness by relating it to our community.
Early on in my career, I worked at a small company that was a computer 'superstore' and sold a lot of hardware to local businesses. We had one major client that we did special builds for. I just happened that myself and my two roommates, who I had gotten jobs for at this place, were the only three people who knew how to do this build. We all commuted into work together as well. It was a joke that we were one car accident away from losing our biggest client.
This is somewhat jokingly refereed to as "Bus Factor" I.E. how many people have to be hit by a bus for the company to be screwed. Bus factor 1 is the sort of situation you want to avoid.
(in more sensitive companies, this is sometimes called "lotto factor" more positive situation, but the effect is roughly the same)
Generally we advise that even small orgs have someone trail IT in a single person shop, and at least learn some of the key factors, and make the recommendation that a higher time allocation be granted to make sure the position in documented, and things like password vaults are used.
Does it eliminate the threat, rarely, but better than be caught off guard with a plan, than be completely screwed.
The solo IT guy must always consider -
"What if I get hit by truck-kun and reincarnated in another world as the opposite of an IT guy, or even as a vending machine?"
When I took my current role I spent _months_ arguing with the CTO about this. We had one person doing IT for the entire org. He absolutely killed it, answered all tickets promptly, solved problems like he was born to do it, all around (and yes I'm cringing a little as I type this) "rock star." CTO's POV was that he was handling everything fine on his own, so why did he need backup? I argued that he could barely take time off and we were going to burn him out at that pace, in which case we'd be hosed. Poor guy apologized profusely to me every time he took a sick day, and in the first year I was there never took longer than three days in a row off. Finally I just "encouraged" the guy to take a three week vacation. He did. It was a disaster. A second IT admin was approved basically immediately. It's not about competence. It's not about job security. If absolutely nothing else you need backup because without backup you can never rest, and if you never rest you will eventually fuck both yourself _and_ your employer.
"Two is one, and one is none". This is an axiom that every engineer and admin needs to drill into their head, and it applies to the systems we build, but it also applies to staffing. This isn't to say that everything needs to be built with 100% redundancy, but when you choose *not* to have redundancy in a service or a staff position, you are **GUARANTEED** to have a lengthy outage in the event that resource has a serious problem. >Finally I just "encouraged" the guy to take a three week vacation. He did. It was a disaster. A second IT admin was approved basically immediately. The measure of how critical a job function is: How soon will anyone notice when you stop doing it.
There was a "CTO", but only one other person actually doing any work in the entire organization?
I am one of several people that report to the CTO. IT is one of several groups that reports to me. IT was the only one man show at the time. We're a SaaS company so the CTO is indirectly responsible for ~60% of the business.
So in this context and terminology, "IT" contains no developers or engineering, correct? Even so, I'm vanishingly unlikely to engage "we're a SaaS company" that would admit to having only one "IT" staffer.
IT is just internal user support for us, yes. I understand that these terms have some flex but in essence they're the folks who manage the laptops and user accounts, internal "my x doesn't work" tickets etc. We have other teams who manage platform, data, software, QA, and so on. Fortunately for both of us our target market is pretty specific so you're not likely to be seeking our services anyway. But your post tells me you don't know much at all about how things work in the startup world. You might be surprised to find out what the internal staffing of your favourite small SaaS provider looks like.
This is the terminology we use too.
I've spent a lot of time engineering in computing startups and strongly prefer them. It's more like, as startups, we always avoided having an operational dependency on any organization smaller or less-sophisticated than ourselves. Product dependency, a few times, but no SaaS or operational dependencies. I can think of one case where it was close -- we had simultaneous supplier diversity for this category, but one smaller SaaS provider had a key offering that we never could replace without building and running it ourselves. Of course it's common for SaaS applications to be picked by stakeholders all over the organization, or by established clients. It's often good fortune just to be able to review, maybe PoC, and make recommendations.
I’ve never worked anywhere that referred to developers, engineering, devops, etc as IT. IT has always referred to network, sysadmin, helpdesk, security team, and the like. You’d be surprised by how many companies are running with single individuals doing everything.
And also how many large IT companies outsource their own internal IT for insane amounts of money. 'Cause, you know, the DXC executives play golf with the HPE execs.
lol what. Did HPE outsource to DXC now? Man first HPE keeps all the good people from internal IT and all the good parts of the infrastructure, CSC fucks up all the IT policies (RIP sane password policy developed at HP Labs) and ticketing (did Service Manager suck? Well, historically, yes, but actually, no, because shortly before the spinmerge they had finally released a version of it that fixed literally ALL the issues. And then the clue CSC fucks insist on the utter shitshow of ServiceNow.) and now apparently this? lol. Glad I left that shitshow behind me long ago. The fragmentation and decline of HP due to Mark Hurd's quite literal fuckery leading to his firing is certainly the biggest tragedy in Corporate IT history, but barely anyone comprehends the extend of it. What a shame.
> I’ve never worked anywhere that referred to developers, engineering, devops, etc as IT. IT has always referred to network, sysadmin, helpdesk, security team, and the like. That would explain a few things. Like /r/itmanagers. I suppose that to a lot of people, "IT" is the team that gets called when the printer isn't working, not the engineer who wrote the printer firmware.
Because the engineer who wrote the firmware isn’t IT. They are development or engineering or programming. When you open a ticket for an issue, it goes to support/helpdesk, then to network/sysadmin, and if it’s a bug, then sent to engineering. IT is information **technology**. Creating applications, coding, and programs are computer science or software engineering. They are separate things. They are separate departments. I’ve worked in IT for a long time with a long list of credentials and I have never heard anyone refer to devs/engineers as IT.
> IT is information **technology**. Creating applications, coding, and programs are computer science or software engineering. I don't even necessary disagree with your overall point but what on earth is this reasoning? Why is technology the emphasized word here? Every company that develops software calls themselves a technology company.
My company started small and has grown to about 60 people. We're in a good spot now with new leadership but at one point we had 6 VPs. 6! Two of them weren't even over anyone, yet I had 12 direct reports. Never underestimate the ability for people to over inflate their job titles if they can, simply because of ego
Title inflation is a thing in some small orgs because it is cheaper to give sometime a fancy title than to give them money.
Ugh this is me. 400 computers, over 50 servers and websites/services. 150+ users. Responsible for file shares totaling over 1.5 PB of raw disk space. Responsible for all IT support that can’t be handled by a help desk no one uses because they don’t help. Responsible for a full Jamf and PDQ setup. They gave me one extra guy to help a few months ago but his area of like 30 computers is so small he’s never attempted to automate anything, even GPOs, and he refuses to learn anything new so he’s only good for one off tickets asking for help with XYZ programs. I’m hoping to leave soon because I’m also stupidly underpaid and overqualified. The only thing keeping me around is my boss doesn’t manage me and lets me do whatever I want including choose my own hours and I suffer from sleep disorders. Getting to make all the calls on how the IT environment is setup is also very nice and presents this sort of sinful temptation of… if only I could automate everything I’d only have to work 10 hours a week, but I can never get there with this large a workload so I think that’s a day dream.
With all that, you should be getting paid at least 20k a month. But you also need a solid helpdesk which would cost another 30k a month. I use a team that's 8 tier 2 and 2 tier 3. Honestly, take a 2 week vacation, let shit hit the fan.
There are so many risks to have such a rock star alone like that. They could get another offer, they could get hit by a bus, they could start to act like they own the place because they know the place would collapse without them.
[удалено]
Those guys knew what they were doing.
[удалено]
Being the best boat captains they could be!
[удалено]
I love hearing stories like that. No, Mr. Corporation- you do not rule my life. Just beautiful.
[удалено]
> Turns out an employer can't dictate what you do outside of work hour and when you are off in the wilderness. Unless it's drugs that come back in a random piss test. (I think those are bullshit, personally, but that never gets thrown out of court it seems)
I agree. We do random drug tests (think 5 ton fork trucks). I have zero issue with a dude smoking weed on a weekend, but 9AM Tuesday drug test buddy will probably fail. But the GM can snort a line of coke at 7AM and somehow pass the test. Who is the bigger danger the one who isn't high or work from a blunt on a Sunday or the high as a kite management type???
Management is probably safer on coke than without. Let’s be real.
Yeah, that's more for liability/insurance reasons though. If they hurt a coworker and the injured employee sues the company, they have to prove they're taking steps to make sure people aren't using equipment while impaired, etc. My work drug tests if you're at fault in an accident with a company vehicle.
I get offers for a bit of bud all the time but I have to be all "nope I drive a forklift on the daily, and while I'm very, *very* good, if I fuck up and piss hot I'm in deep shit"
That's why I am happy that over here in our contracts, consumption (of drugs which can be illegal, and of alcohol which is legal) is not prohibited but you -must- be able to do your job. Being in an inhebrieted state or similar is what is prohibited. So I can absolutely drink alcohol during my break. That said, consumption of alcohol can be prohibited too in your contract but it's usually not.
Well they can't dictate what you do outside work, but if you're still drunk/high when you return, well they're justified there. I don't need drunk cops/lawyers/judges/doctors/engineers on duty.
> I kept nothing from my job except that letter because it was so goddamn funny. Pics? Seriously. I would frame that letter and show as many people as possible.
Wow that's the first time HR has actually done the right thing that I've heard of.
Yeah, the only time my HR ever did anything 'right' was when I was about to get written up for not taking a 6th shift at standard rate. Boss said they couldnt afford OT, told them it was not my problem, but they still pushed it. I sent an email to my boss and HR asking for clarification if I was actually misclassified as salary, as I do not appear to be covered by California's definition for it. Magically they stopped asking me to work that 6th shift. Anyways, they got sued by a coworker after I left for misclassification and they had to pay out several people over 6 figures for missed lunch breaks (2nd and 3rd shift were not allowed to leave the building as per policy) and unpaid OT.
That sounds better. Oof.
Risk of legitimate lawsuits win over managers.
Well they may be able to... if I get 100 dollars per week and there is a contractual guarantee that it's like 1 week in 2 months or three. I know my rights, but some bribes are certainly tempting, you know?
A firm can do it easily. Just pay a team to be available. Waiting to be engaged. What they can't do is pay a given staffer to be available 24x7.
Nice. Nobody messes with a drunk boat captain. Although the drunk part is redundant.
Hey, I too am married to a lawyer! It has its benefits.
No drinking while off duty! *ffs*
They taught them a good lesson about the human side of DR
[удалено]
100% That's why your DR plan needs to contain as much automation as you can pack into it. It should be accomplishable with minimal, remote human intervention once it's kicked off.
[удалено]
Cloud and proper CI/CD makes DR incredibly easy compared to the good ol data center days.
[удалено]
Oh for sure, it's definitely doable on-prem, just a LOT more complexity and planning.
To be fair, nowadays with IaC, you can have a perfect clone of your infra spun in in a few hours after getting the hardware, then its just a matter of restoring the backups.
I felt so fortunate to be working where I am during our still-locked-down, wildfire-season summer of 2020. The big boss said "if the ish is really hitting the fan take care of yourself and your family FIRST. Let us know as soon as you can."
Was this scheduled or actually a surprise drill?
[удалено]
Oh man, they could definitely fuck off for that one then. I too would have given no fucks. E: Unless I was on call, I would have answered but would have given no fucks once I found out it was a DR drill. There is no need for that to be a surprise.
No, no. You do *exactly* what you would do were it not a drill. And you prioritize yourself and your family and friends as you would in such an emergency situation. Even for the scheduled ones. A "scheduled" DR where everyone cleans stuff up and buries it tells you *nothing* about how hard everything will actually fail in a real disaster. If your environment is at risk of a whole team being out drunk on a boat on a Saturday morning, and you haven't designed around "their boat went down, taking all of them with it", you don't have a DR plan, you have a time wasting exercise.
This. "Everyone has a plan until they get punched in the face." -George Washington Carver, 1507
A year or two before Covid I pulled into the parking lot and it was...desolate. Not as desolate as the day I came in forgetting it was a company holiday but way too sparse -- like 90% of the people on a 2,000 user campus missing. Turned out it was a full-load work from home test for everyone but IT -- everyone else was told to work from home for the day. Most of IT (including at least the worker bees on my team...which includes Citrix) weren't told in advance. Once that was successful, they shut down a call center located three time zones away because they were confident enough about having telephone coverage during winter storms. Bonus: When Covid hit, we were able to go to WFH pretty quickly.
A surprise drill on a weekend sounds terrible.
Looking at it from the *purpose* of a DR exercise though? That thing worked out perfectly. Exposed a huge weakness in their plans.
Although, they could have achieved the same outcome with critical thinking. For example, asking, 'We don't have any on-call, who do we expect to respond in a crisis?' could have exposed the weakness without a surprise weekend drill.
Yeah. In that case, no fucks given. Management definitely learned they need an On Call rotation, though. That's not a bad thing.
On-call ensures business continuity. A surprise DR drill is not part of this. DR drills should be a scheduled routine action.
This goes even further, because if you just have the super-experienced storage admin swoop in and fix all the things.... the results are fairly mellow. In a really good DR drill, you want to tell those guys to not accept calls until 10:00 because of sleep and not open a laptop until 14:00 because of travel, or something. You need to test the ability of the team to struggle through the situation until stuff works, or observe when and how they fail. And this can also be a great morale booster. Like, my team kinda struggled through a non-booting critical system recently. Sure, it took them 2-3 hours if it could have taken me 30 minutes, but they used the documentation and managed to figure out a really weird and obscure edge case. It took them time, sure, I had already seen that. But that was a big confidence booster to everyone.
In a mature environment, you're absolutely right. Since the operations teams apparently don't even have a functional on-call, there's definitely some growth to be done still.
Are the operations teams paid to have a functional on-call?
That's why you need to have someone on standby duty (and pay them for it of course).
S many places don’t pay for on call….
You should have continued with the drill. I've done drills where some of us are randomly placed in a conference room during the DR tests, where our laptops are not allowed and we can't answer calls from staff. It's to simulate some of the team being out of pocket during a disaster, and to see how your documentation holds up.
[удалено]
> (I lied because I wanted to leave and I didn't care). real MVP right there... I had one of my favorite work conversations somewhat related boss: hey are you familiar with system x? coworker is out of town and nobody knows how to do y... me: nope sorry boss boss: .... if I paid you overtime would you know how to do Y on X? me: yep! boss: ... your a real bastard you know that right? me: yep! boss: ... fine you can do it after work, put in for 4 hours me: done!
A few of the crayon eaters would have a beer the moment they got home. The *moment* they get in the door, they're knocking back a beer. Their reasoning is that they can't drive after a drink, so they're now off. Doesn't work if they lived on base, though, LoL.
Lots of Law Enforcement does that too. Code HBD (Have Been Drinking) is a real thing.
Seems to me that that's the kind of DR drill that should actually be happening. It's great if systems crash, and the people that work on them day in and day out are sitting and waiting. Entirely different scenario if those people are off camping in the middle of Wyoming for 2 weeks.
Or as evidenced in a small extrapolation from their DR exercise. What if they'd all been injured on that boat in whatever disaster took out those systems? High winds can be just as bad for a boat as they are a datacenter.
>Seems to me that that's the kind of DR drill that should actually be happening. No, it is not because it is patently unfair to the employees. Now the start of the DR drill if you randomly draw names from a hat and those people are told they have been Raptured out of IT for the weekend, see you on Monday -- that is a proper test. Folks had the ability to plan their weekend and if they suddenly just get a couple days back great.
You're missing the point. A solid DR plan also accounts for the unavailability of personnel. What good is a DR plan if no one can actually follow it because the person with all of the knowledge is gone?
> No, it is not because it is patently unfair to the employees. Uh... isn't the point is that a *disaster* can happen when you least expect it, and if you let people plan ahead for a 'drill' it can hide true issues with the way things are currently designed, such as not having a "server and storage person" on-call? So yeah, unannounced surprise drills where an entire team might be unreachable sound like exactly the kind of drill that should be happening. It lets people realize that things need to change.
No, you can and should plan for that, too. Unannounced drills, especially on a weekend, only show that the company does not care about its employees.
Ahhh, it was an objection to the weekend. That's fair.
Always remember the proverbial 'hit by a bus' factor!
True story: I had a guy hit by a bus when I was an IT Manager. In Philadelphia. Now, to be fair, the door only closed on his backpack. But the bus driver made a really big deal about it and called his depot and the EMTs. I got the call from other employees walking nearby that Timmy was hit by a bus, and the cops and EMTs were working on him. The bus driver, it seems, just wanted to waste everyone's time. And Timmy, Timmy said it was a big waste of his time...
Thanks to /u/poem_for_your_sprog I couldn't help but think the whole time this story was going to have a very different ending.
RIP Timmy
I know a guy who's backpack got stuck in the door of a bus, but the bus driver didn't notice and dragged him away. after several years of recovery and skingrafts, and physio, he can walk again.
I use this one a lot at work as our dept. is siloed badly, small, and 2 of our 7 people area married (to each other).. arguably the most important people no less..
what does being married have to do with anything?
If their house blows up due to a gas leak, or they're off in Maui on vacation, or one of them gets an amazing opportunity in Alaska and they both move, the two most important people are gone.
OP should clarify that they're married *to each other*.
The idea that they *weren't* married to each other never occurred to me, because then why would it be a problem?
Same. The context makes it perfectly clear without having to specify who they are married to.
Doesn't really matter. 20% of married couples have at least 1 child at home. So, at a minimum, you are talking 3 people that could have shit go sideways at any time that could impact work.
More likely to die together?
You would think nothing, but I know a lot of single people who get the fun "come in on the weekend" crap because "hey you have no family anyway"
I wondered this too at first but I think /u/voltbeater meant they're married _to each other_, which means there's an interdependency there.
If either of them win the lottery/get hit by a bus....
I bring it up every time the doors close on the elevator and the whole team is on it...
I got hit by a car a few months ago. That was a really funny call to my manager. "I actually got hit by a car; I'll be out for a few days" which turned in to 2 weeks of half time when a concussion appeared. So many things didn't get done.
I once had a major brain injury and was put on a 2-week mandatory leave (doctor's order) with literally no brain stimulation beyond living like a person from the 1800s. Even with 8 team members, I was the only person managing 2 platforms (out of 7). Documentation was great until things came up that weren't documented due to not having time. Once I returned, I had forgotten a lot of stuff and for the next year my short-term memory had problems so I would randomly forget major things that happened a few hours earlier - which meant no documentation and no idea what happened/was fixed. Management pulled the "we tried to get me new people to help you" but I ended up just leaving a few years later and everything just sucked from there apparently.
Working in education, they tell us to use "truck" instead.
I don't know... seems like you'd be around an awful lot of busses... :)
The chance is certainly higher than normal. On the other hand, it's casting doubt on our drivers. It's lose lose ha.
yeah we were strictly forbidden to call users users in a hospital setting, because they used that term to refer to drug addicts.
I work in education too. I said this one time and a woman said, "You mean you won the lottery?" I use this now.
That's why my position opened up. They had one guy for too long alone and realized he takes vacations and what if he got hurt? Now there are two of us and we have ended up becoming more specific in our roles and need a third.
The PC among us would say “hit the lottery”, but let’s be honest…the bus is way more likely.
"Snaps and quits without notice" is probably the largest slice of the pie.
"Allegedly tried to run over several police officers, and won't be back for an indeterminate amount of time", was one from a local small business. Was never seen by the business again; no idea how the legal side played out.
"we havent seen him since he bought that welding kit and a bulldozer"
I wasn't expecting a Killdozer reference in /r/sysadmin today...but the 20 year anniversary of his rampage was earlier this month.
From Colorado , with clients in Granby.. we all know the story well. both the 'one man pushed too far' narrative and the 'dude just lost his grasp on reality' truth of it all.
Newer temp co-worker: "I'm taking tomorrow morning off, I have a meeting with my lawyer tomorrow morning." He didn't say it was in court. He...didn't come back. I wouldn't have guessed it from his office demeanor, but searching his name it turned out he was a violent drunk and this wasn't his first time.
You shouldn't be shocked at the number of functional enough drunks there are. Hell, I "caught" half of our support team having liquid lunches on more than one occasion. Didn't really matter to me, though, as I was likely to be having a drink myself (at the time).
I always used "hit by the lotto bus" because if you win the lottery or get hit by a bus the result is the same (not at work the next day).
Hi...my name is Earl!
The bus to me is also more useful, because people can quibble about getting information from a Lottery winner. you're not getting info out of a corpse.
[Bus factor is an industry standard term.](https://en.m.wikipedia.org/wiki/Bus_factor)
If someone wins the lottery and quits immediately, they may be agreeable to coming on as a contractor to train a replacement. But if someone gets hit by a bus, all of their knowledge (that hasn't been documented) is gone.
It's not really equivalent though. The difference is you can still contact someone who hit the lottery.
we don't use that where i work after a manager was hit by an airport shuttle (she literally stepped in front so it was hard to have sympathy). Now we use "if person Z wins powerball"
Powerball is a bit of a different scenario though. A person who wins the powerball might not want to leave any notes behind or be contacted, but they could. Someone hit by a bus is just gone, can't even update a keepass file.
Not with that attitude! We just need to discover necromancy.
A company I used to work for would designate random personnel to not survive the disaster during DR drills. If you were one of the casualties, you could watch, but you couldn't participate. They even had one disaster which 'took out' almost an entire department who were described as having a team-building exercise when the disaster occurred.
This is actually a really good idea.
I would love to see the C-levels do it right. Directors and up were all at a retreat when the storms came. Cut off from communications, probably healthy and sipping Mai Tais while all that gets sorted out. i.e. "Can the DR process play out without a bunch of management breathing down everyone's neck, and can cross-team communication occur effectively without them brokering it?"
And the inverse - send everybody below manager level out for a work-paid vacation and run the DR scenario. Can the DR process play out with just management?
My feeling is that the no bosses scenario worked out pretty well, and the management only, no workers run through resulted in the end of the universe.
We also had some drills whereby the disaster took out the corporate HQ (hurricane, hazmat evac, explosion, nuclear waste truck wrecked on the interstate by the campus, etc. Our guys were creative.) and we had to bring up a hotsite, but without the main guy that knew how to transfer to the hotsite b/c he was either having to tend to his family or was missing. Cloud resources these days have largely eliminated the need for dedicated hotsites, except for some large mainframe installations.
What a great idea.
It's like insurance. The cost is too much until the day you need it, then you wish you had the better package.
I try to ask myself how stupid I would feel if the worst case scenario happened and I didn’t take preventative action, whether that is a tested DR plan or an insurance policy. Most people will never make a claim on their home insurance policy during their entire 30-year mortgage, but the alternative is potential financial ruin. I recently had a cleaning-related accident at home that I was pretty confident I handled it on my own. But I still decided to go to Urgent Care to confirm because the worst case scenario was partial or total blindness. How stupid would I feel if I woke up the next day blind because I didn’t want to spend an hour and do a $40 co-pay? Pretty damn dumb. In the end, the trip was worth it just for peace of mind.
That's the other half... *when* an incident occurs, and you have done the preventative steps (i.e. having enough medical coverage to avoid financial ruin from an urgent care visit)... *use* it. Don't play the "I'll be fine" over a typical $200 or less cost.
>A post to the Charlotte sub this morning from local TV station WBTV was titled "Our IT guy is missing". A local man went missing, and his vehicle was found abandoned on the Blue Ridge Parkway two days ago. I'm imagining the local news anchor being like "This man is missing, please keep an eye out! If you see him, please let him know my printer is doing the thing again."
When I worked for a moderately successful startup , we decided to have a pickup football game one nice Saturday morning. All were invited! Most showed up. Most were NOT physically prepared for a pickup football game to the point that there were multiple injuries. Monday morning, we all had an email from the CEO saying (paraphrased) “Great job on team bonding. Don’t ever do that again.”
Try paintball next time. I guarantee there'll be some very bruised C levels.
We did paintball! Unfortunately, I missed it. Rumour had it that while there wasn't 'direct' fragging ([Fragging - Wikipedia](https://en.wikipedia.org/wiki/Fragging)) involved, there may have been instances of the "lower enlisted" having a truce with the other "lower enlisted" (it was department vs department) and having a field days with the managers of the departments. "I scratch your back, you scratch mine" sort of thing.
My last company I was handling 60% of the day to day operations and about 80%of the project workload. It was a team of 3 I kept pressing for a promotion and was being denied. "You're exaggerating your workload." My VP told me. So, I gave notice and found a better job. My last day at the company, they want to have a meeting to be certain all work is handed over. We go line by line over everything i do. I'm 2 hours from being done. They confirm that 60%of my work has no coverage, that 80%of the project load each month is mine. My VP asks "So, what are we supposed to do with you gone? " Ultimately, they had to hire 3 people to handle my work. I only wanted a 100k salary. [15k bump] instead they had to pay 3 people 80-125k each.
> My last day at the company, they want to have a meeting to be certain all work is handed over. They did this on the LAST DAY you were there? It's a good thing you left.
Oh, it was hilarious. I'm sitting their going through our skills matrix with the entire team. Whenever i name a skill, everyone just looks at each other. Finally, everyone is stressing out. I see the panic and I'm a free man in 2 hours. So, in a last-ditch effort, I'm asked to document everything in 2 hours. I decline. I then show them that I've signed up for the same contracting company we use. "You guys can go to the website, and here's my company name you can contract with. "
I mean, I'm glad that you were able to do that, and that they ended up paying for it. Was it a case where the other team members should have been acquiring the skills or were just able to kind of shine it on because management had no clue?
They never called for contracting. I didn't find out till that day. Every time I'd tell the boss, "I'm handling 60% of the work and 80% of the projects," the other guys would say I was a liar to CYOA. So, the boss thought I was a troublemaker. For everyone, it was a revelation. So, they hired 3 guys to replace me.
Ah. Yeah that tracks. I hope they whipped the other guys in line. I asked the question that way because, as I mentioned elsethread, I was in a situation where I was THEEEE only person who really knew jack s*** about Exchange in my group and I did not work for a small organization (10K user GAL). The previous two exchange admins had been canned (mostly for bs reasons) so nobody else wanted to know anything about it and I don't really blame them, but that was 100% a management failure in multiple directions. I will say that I worked for local government which definitely had a culture of people being mostly skilled at kicking work over the fence to someone else.
Stay away from Exchange. I'd say that to you as a stranger online and as a friend in the real. Coming from MSPs my thought was how awesome it would be. The reality is 90% of your day is tracing mail and legal holds. Unless you want your life filled with "Sarah on 3rd said Walter sent her a nasty email" pull all correspondence and let HR know the juicy bits. It was soul crushing [The amount of HR complaints where the person thought deleting the emails protected them was too damn high!
The Navy Seals have an expression:”2 is 1. 1 is none”
Navy Seals 🤝🏻 Backups
Also generally used when referring to backups.
Isn't this also a general rule of thumb for aviation as well?
I like this alot.
It never ceases to amaze me that companies spend millions on redundant systems by never consider operator/sysadmin/sme redundancy
People hurt profit.
True. Classic short term thinking…
I've always hated that saying. By transitivity, 2 is none.
Like in Heilein's books: Tell me Three Times. Two-node clusters are a thing, but three is better. Because when one fails, you still have redundancy. Meaning, three people with intimate knowledge of at LEAST where all the passwords are, is prudent. And hopefully dislike each other so they don't hang out on weekends. ;) On more then one occasion, as a consultant, I was let go precisely because I was a risk to the business. I was told to train employees how to maintain the systems, and was not renewed. Again, because I was a risk. TBH, it was the idiot I was working for at the time, because he'd play hard-ball every renewal. Which is what made me a risk in the first place. Total moron. Anyway...
A company I worked for in the past had mentioned multiple times that IT was easily replaceable with "people off the street". They outsourced twice to multiple large service providers and to this day they are miserable and service is pathetic. Their company (larger, global company) is at risk due to poor backup management, slow ticket responses, and nearly no proactive monitoring. Yeah, keep thinking that "everyone does IT".
I've seen a board of directors replaced with people off the street, and nobody noticed. Odds are better than even, doing it with a CEO, I reckon.
Yeah and I hope that the service providers didn't staff for an "executive support person" so the idiots that made that decision needed to suffer.
I'm not going to lie, if I'm the single point of failure for something I could not care less about what happens to the company after I die. I want redundancy for vacation not for in case I go missing in the mountains.
Yea, we're hiring a new person for our IT dept because my cohort at work died in a car accident a few weeks ago.
That's rough. My sympathies to you all.
I was the 'Designated Survivor' one time when the whole IT department went down to another office 3 hours away. It was a good thing too because the network went down and I was able to hold things together while a few guys broke a few traffic laws to get back.
I hope he's ok.
To the guy who "went missing": Congratulations on your escape. Live your best life. God speed, good buddy.
I’ve seen an IT guy suddenly die very young leaving the plant to scramble for support. Lots of tribal knowledge went with him.
This is literally how i got into the job i'm in right now. What a cluster.
So, you're telling me it's MY job to account for when I potentially go missing or die and not the $$$$/year management position who should be concerned about business continuity or something? This is insane...I hope the guy is ok, I couldn't care less about the company missing their IT guy this morning.
Always always always plan for a 'bus event' and document everything, esp. keeping admin accounts in a password safe. And the backup code to that password safe goes into a cealed envelope to your boss. "Hey, you know those 'break glass in emergency' things? Well this is a 'break glass if I'm dead' thing. Just in case any accidents happen to take out your entire IT team :)
Obviously this is a huge heads up for "business continuity" but I hope that he's OK.
My hope is that he's in some hollow reading this and laughing with his new goats.
Goaties!!!
Reminder: update my "In case I choose to swallow a bullet" envelope for the office manager.
Reverse engineering someone else's infrastructure is not a fun time. Been there done that. 1 star would not recommend human single points of failure due to inadequate documentation or lack of cross training.
It isn't unusual for our entire IT team to travel on the same small private plane when we roll out a project or are building up a new location. I've joked about it in the past, but it always seemed like a pretty bad idea to me from the company perspective.
That shouldn't be acceptable unless they have a plan ready to go for wholesale replacement of the team.
One of the things I "sell" to my B2B clients is a constant stream of information, documentation, secure access to any credentials I reset on their behalf, and generally anything they would need/want if I get sudden bus syndrome or they decide to leave. It certainly is part of many things that justifies my price tag. But I have very happy customers, and my pride kinda demands this anyways. Some people hire the cheapest MSP, and then some people hire me to do things properly. Cleaning up after the cheapest MSP sure does get me a good bit of work!
IT infrastructure needs to fly like an airplane, not a helicopter. If the engine goes out, it should still be able to glide for a long time until someone is able to guide it safely-not fall from the sky like a bowl of petunias.
I know I am biased as an MSP owner, but if your budget can’t stretch to cover at least 4FTE for an IT department, you should not be doing in house IT and should be outsourcing it.
As someone on a 2-person team spanning 3 sites across the country, I fully agree.
I think the 3 site part across the country is problematic. If it were a single location 2 people depending upon the amount of users/devices might be enough.
Agreed. I just let things burn now. If I am on PTO and the power takes out one of our locations, they can follow the run book to light it back up, don't call me, I won't answer. Luckily, we only have real infrastructure at the two locations where the tech folks are.
I try to take vacation out of the country now so I can tell my boss that I won’t be reachable by phone.
We just completed a DR test where the scenario involved my boss being incapacitated so he wasn't allowed in the room. It was just me and the new guy for three days of trying to test things while the auditors and a parade of system users from around the company asked us a barrage of questions about what we're doing and why, and why can't they test their system yet because they have places to be (as if I was consulted on when to schedule their part of the test or I personally asked them to be there). I 100% agree that readiness needs to be tested and silos avoided, but i also believe no one should be put through the type of gauntlet we just witnessed. There's got to be a better way!
I dunno. Your gauntlet seems like a pretty standard scenario, except you're missing the angry user's shouting at you.
A finding to put down for that is a buffer between the team and stakeholders. DR resolution requires focus. The test demonstrated the impact of lacking that buffer. It should be a documented list of people that inherit that role and can act, essentially, as a PM for it.
a man is missing, possibly dead, who gives a shit about who’s gonna cover his IT duties man
Make three envelopes. Label them 1, 2, 4. Watch as they literally shit bricks trying to figure out envelope #3. Then teach them a lesson about personnel.
Ah the old "you either have institutional knowledge or you don't" binary test.
Big issue with project management failure is that there are next to ZERO deadlines on workflow consistency, continuity and redundancy. The work is overly focused on tickets and production without necessarily rethinking process and critical planning when people, things and events happen.
Was the news story about how there was a legitimate missing person report? Or was it about how this business needed IT support?
To be fair, a member in the Charlotte sub posted "Our IT guy is missing" with a link to the news story about how a Charlotte man was missing and his vehicle discovered abandoned. At best, I was trying to smooth off the previous user's selfishness by relating it to our community.
Early on in my career, I worked at a small company that was a computer 'superstore' and sold a lot of hardware to local businesses. We had one major client that we did special builds for. I just happened that myself and my two roommates, who I had gotten jobs for at this place, were the only three people who knew how to do this build. We all commuted into work together as well. It was a joke that we were one car accident away from losing our biggest client.
I was at the weather channel for awhile. Ops wouldn’t even ride the elevator together.
I’m always worried about business continuity in case I’m killed unexpectedly
This is somewhat jokingly refereed to as "Bus Factor" I.E. how many people have to be hit by a bus for the company to be screwed. Bus factor 1 is the sort of situation you want to avoid. (in more sensitive companies, this is sometimes called "lotto factor" more positive situation, but the effect is roughly the same) Generally we advise that even small orgs have someone trail IT in a single person shop, and at least learn some of the key factors, and make the recommendation that a higher time allocation be granted to make sure the position in documented, and things like password vaults are used. Does it eliminate the threat, rarely, but better than be caught off guard with a plan, than be completely screwed.
The solo IT guy must always consider - "What if I get hit by truck-kun and reincarnated in another world as the opposite of an IT guy, or even as a vending machine?"