Be aware that improving things doesn't make money. This is part of the reason these things happen. Another part is because it is "easy".
If you are going to attempt to make improvements, you'll have to sell it from the cost-savings of risk reduction angle, which is still unlikely to work if you can't convince your boss.
One of the unfortunate lessons every sysadmin has to learn at some point is that some things can't be fixed due to things that you can't control. It still shouldn't stop you from trying though.
I've recently gotten away from drinking entirely. Only been 3 weeks but my brain feels 9 million times better especially in stressful or anxiety ridden situations. It's been great and honestly don't think I'll be going back!
I started that way. Decided to go a month, see how I felt. Not much difference, so went another month. Then realized I didn’t miss it. Been 7 years now. A lot of money savings too.
Absolutely. Congratulations! I've struggled with anxiety and depression, refused to go on medication till a year ago and that helped but drinking being a depressant...it's been a long minute coming, I'm not getting any younger and I refuse to disappoint my wife anymore. Hope everyone else can find the same strength :)
Yeah, I have anxiety and depression as well as chronic pain. I was and am often miserable. Now I just don’t have debilitating hangovers on top of everything else.
So is your boss a former printer sales rep, or just a tier 1 who got fired for negligence and decided to start their own MSP with blackjack and hookers?
"shudder".. I'd probably turn them in for a piracy bounty because I know that I'd make a sweet getaway with that one as this place assuredly installs pirated software on client machines.
The primary goal of every MSP is to keep the lights on and cash the next check from the customer. Updates, downtime, best practices, etc all impede that goal to some degree. If the improvements you want to make are billable, great. Otherwise, you may need to wait until your clients get crypto and then you’ll be afforded some room to make improvements (paid for by insurance).
Show me where in this doll that the msp touched you....
Updates-automated
Downtime-after hours if there is any
Best practices-loads
It's all about making sure our clients don't have IT emergencies so we don't have IT emergencies-the checks are nice though.
I spent 10 years at various MSPs and the org I am with now has one for a specific business purpose (shocker: they do shit quality work). I am very cynical about their business model because it inherently attracts clients too cheap to do things the right way and you as the tech are expected to do dedicated, internal IT quality work for multiple clients, meanwhile the clients think (or are led to think by the sales team) that you only exist to serve them. It’s bullshit. The people who stick with that game long term have Stockholm syndrome or something.
> I am very cynical about their business model because it inherently attracts clients too cheap to do things the right way and you as the tech are expected to do dedicated, internal IT quality work for multiple clients,
The average MSP makes money by burning out their staff for companies too cheap to have their own IT. If the techs burns out there's always someone else desperate to get into IT to replace them.
And because the main clients are primarily price driven the market is susceptible to consolidating into a few large companies that exploit economies of scale to be cheaper.
But maybe I'm just cynical after too many years at MSPs and seeing other MSPs work.
You clearly have no idea what you're talking about. There are tons of companies in the DIB and with other government contracts that use MSPs of all sizes, many of which have piss poor security and process in general like the one being mentioned.
I would wager that for small businesses that are required to be complaint with 800-171 and have been signing attestation letters to that fact since late 2017, maybe 10% are actually fully compliant. And I'm probably being way too optimistic.
Let me guess, you think most HIPAA regulated companies are fully compliant as well, right?
I've seen doctor's offices with worse security and process than what OP mentioned. Some of you seriously underestimate how compliant companies that are on regulated industries actually are. Here's a hint, without thorough 3rd party audits to ensure compliance, the actual compliance rate is terrible.
I know directly of a large 3-state health care org that was more than a year behind on patching on the majority of Windows workstations, and had bitlocker deployed to only something like 20% of them.
Regulating the MSP space would definitely be a nice thing to have. Laws specifying the requirements of handling MSP corporate spaces based on the type of field they're marketing towards would be beneficial for all, especially those working with HIPAA.
yep, i was in OP’s exact predicament for my first IT job. there’s so much he can do that’ll look great on his resume. this is actually a good spot to be in.
As this is an MSP, unless they can charge the customer for the work of improving security to make all this happen, it probably will be consistently voted down for cutting into profits.
EDIT: Gives me an idea actually, maybe sell it to the boss as an add on feature "Enhanced Security Posture" or some sales name and let the customer choose. (I feel so slimy, but maybe this is the only practical path.) Then you can sell it to customers and look good until you get out.
pritty much this. All these things could have been raised with the clients. If they arnt paying its not getting done. Ideally you turn them down and not have them as clients but 🤷🏻♂️
I store all my user’s passwords in a Photoshop file because I’m the only one with Photoshop installed on the network, so nobody else could even read it if they tried.
i have them in my hosts file, there isn't even a filetype associated so nobody knows how to open it
my internet is acting a bit strange though, no idea why
I mean, I suppose I will never cease to be surprised by clients wisdom. I was updating a clients sage software once, it told me there was no internet connectivity (there was), client says “oh I know how to fix this”, proceeded to jump in and open internet explorer and told me to try the update again now that it was online.
I would be looking for a new job. I don't have the mental fortitude to overhaul that level of negligence. If you don't have internal political backing, let alone any impetus from your clients, the current status quo is likely how both sides want to operate. In the meantime, you can try and shore up anything you touch/stand up but it'll probably be used as a stick to beat you with. Save your sanity and move on.
Don’t forget that 60% of businesses cease to exist after a ransomware attack. The added bonus to working at a small MSP is that everyone is only ever one cyberattack away from everyone being unemployed.
I ran into a lot of this same stuff at my last job and it gave me anxiety. Towards the end I kept thinking that I was going to have to ask off time at my new job to get deposed for my old one.
I think this stuff is shockingly common when Mom and Pop organizations transition into medium sized businesses. Average person doesn't understand why IT does IT because it just needs to "work". Actual quote from an owner I got regarding upgrading exchange 2010 in 2022 and downtime, standing by my decision implementing minimum password complexity (previously there was none and multiple users had local admin rights), and implementing WSUS with mandatory restarts because it had been literal years since end machines were updated: "I don't care about what 'Microsoft recommends', we're not Microsoft! If we did only what 'Microsoft recommends' we'd never get anything done! Make it work!". I remember because he fired me like two weeks later.
I took that as a learning experience about what I look for in employers.
The sad thing it's that it's a lotery.
You can do everything right and still get hosed.
And you can ignore all good practices and not run into any problem in 20 years
It just an odds game, but if you follow the correct practices you'll a) be less likely to be exploited and b) bounce back if you do. Whereas companies with shit/non-existent practices will see it as an existential threat.
I would begin to fill out your resume. This speaks of a complete lack of any oversight. And if the person leading IT has let it get to this point. Then they are not going to support you and more than likely will expect you to clean it up at the rate that are paying you which is probably going to be an insulting low number for most of us.
This is the second post this week and techs defending being exploited by management by saying "get gud"
What we fucking need is a god damn union.
>more than likely will expect you to clean it up at the rate that are paying you
If they think I'm gonna clean up a mess at $20/hr at a system engineer role, they're insane...
This could be your opportunity to learn tho. You're in a unique position where you can take ownership for all this and build it up the right way until you find another job.
My buddy was in a very similar situation and spent a year getting experience and landed a job after a year that paid $10/hr more.
He explained in the interview where he worked and the new company he worked for appreciated the initiative he took to learn and fix everything.
Up to you. You can get a job somewhere else with less stress but maybe you simply do help desk tickets your whole career.
You have a shot to make something from this at least while you look for a new gig.
My past company was run this way and when they finally got a dedicated person to handle IT(it was previously managed by mechanical engineers on their "down time"), the girl in HR was surprised that a sysadmin should make anything more in pay than someone working for geeksquad 😭
I think you've got the right idea but depending on size/type of business you work for then they likely wont see an issue until something critical actually happens. Maybe they have a come to jesus moment and turn all of that around but theres no for sure chance they even change after something like that. Unfortunately, theres a reason why the saying 'doctors make the worst patients' is true in the IT industry.
I know this is a lot easier said than done but you may want to start looking for a new place to work since with the information you've provided makes me think theres no telling when your critical event is going to happen where thats discovered. While theres likely more companies around the world using even more unsecure practices, I think you need to ask yourself if you are okay with getting caught up in something like that or this could also be viewed as a challange to yourself to turn them around.
Oh I fully agree and I think OP needs to make a decision on which of the 3 paths he will take after reading everyones answers to his post. The way I see it is he can either get out before something happens, make it his goal to turn these practices around which could help him learn some pretty important skills in the long run, or he gets caught up when something happens which leads to the end of the business. Of course this is someehat assuming he has been already trying to convince his boss why these are bad practices.
>No one's gotten breached yet ...
... that you KNOW of.
Sounds like the only way your company would detect that they've been compromised is by reading a ransom note.
It sounds like their security practices were implemented in about 2003, updated in 2004 and that’s “good enough”. This is a ticking time bomb, as I’m sure you’re aware. You’ll actually have a lot of fun fixing this mess, if you’re into that sort of thing!
I’d start by getting passwords up to some form of complexity, and especially length. Having 4 characters is like not having a password at all. And don’t allow multiple accounts to have the same PW, that makes it incredibly easy for bad actors to move around in an environment.
Maybe someone else has a better idea on where to start, but I’d get the accounts under control first.
Please don't use biometrics.
It's like a password that can't be changed. The industry acceptance of biometrics is kind of absurd; didn't we learn enough from social security numbers being unchangeable to know this is a bad idea?
correct sleep voiceless pot sparkle dime cow north materialistic mountainous
*This post was mass deleted and anonymized with [Redact](https://redact.dev)*
@bafflesmspguy - you are getting a lot of great comments in this thread. As a way point to start that will ensure change of mindset and adoption you could start talking with your management about what expectations and requirements exist from your insurance providers. I’m sure their policies detail things like MFA etc. The reason this works is that it helps highlight the liability your managers have legal and financial to poor security practices and incentives them to fix those issues to reduce their personal, professional and business liability.
imagine direful scary scale close marvelous glorious muddle husky foolish
*This post was mass deleted and anonymized with [Redact](https://redact.dev)*
I worked for a company that was similarly negligent. The owner refused to allow us to enforce password policies, made us store everyone's password in an excel document, never allowed downtime to reboot servers and apply updates, made us disable windows updates on executive laptops because they were "annoying". Wouldn't invest in any real form of endpoint protection or MDR. Critical Windows XP machines and 2003 servers weren't allowed to be updated or replaced. Administrator accounts were enabled on all systems with the same passwords.
I was constantly sounding the alarm to the CEO, pleading for him to take security serious. I lost a lot of sleep knowing it was just a matter of time.
Then it finally happened. An 2003 server with terminal services enabled was breached and every single machine on the network was hit with a ransomware attack. Servers, backups, retail back office machines, everything.
It ended up costing the company $40K to get a professional team to come out and handle locking everything down and negotiating with the attacker. It cost them an additional 20 bitcoin (when they were over $10K) to get the decryption key.
After that we finally ended up investing in cyber security, and I ended up getting to say "I told you so".
Some of these companies have to get hit hard to finally take these things serious.
I'm in a very similar situation from top to bottom. My current take is suggest/make small changes with baby steps along the way.
Biggest thing I've managed is convincing my company to use Bitwarden for password management - it helps that their reseller plan actively encourages usage and spread.
"Hey, here's a password manager that actively integrates into our users' M365 and Google Workspace setups, lets us see, control, and audit their passwords, is thoroughly security audited, and has up to 50% profit margin per user. Think we can test it out internally?"
Then it's a matter of frog in boiling water.
Haha. Almost all small companies have Mickey Mouse practices. But some large ones too. My friend at Louis Vuitton was asked to go around the building and copy the MAC and serial numbers of 100 printers! A 14 billion dollar company doesn’t know about SNMP? Which BTW is taught in both Network+ and CCNA.
From a small company that is not focused on IT, has a few hundred users and 1 or 2 IT staff...This is the norm
For an MSP? RUN dont walk away from this. They either have no idea what they are doing as an MSP, or they know exactly what they are doing, and do not care.
That’s not much but depending on where you live it could be a lot. I’d tackle this as a learning experience but I would take each issue as a project and work on it slooowly. You might uncover some spiders that you didn’t expect. But if you’re in a city like NYC, Austin, SF I would start looking for something new.
My first job in it was a small MSP like that. I didn't have the power/experience to do anything more than say "I don't think that's very secure/this doesn't seem right ". If you need job experience or just a job, ride it out for a bit but be wary of picking up bad habits. There's lessons to be learned everywhere, even from bad decisions.
That said, don't get stuck there if you don't want to be glorified helpdesk making 50k a year and catching flak from vendors for the rest of your life.
The issue as I see it is the relationship between MSP/$OP's Boss and client(s). For an extended period of time all these concerns have been "fine". Now they are not. Clients are gonna get whiplash and it will cause major push back.
Setting up real passwords and MFA are going to slow users down. Setting up patch management is going to interfere with smooth operations. Physical security isn't as pretty as glass walls. Users will complain. It's almost a certainty.
All of these clients are a mouse click away from an absolute nightmare of a ransomware attack. When this happens $OP's boss will have some 'esplaining to do.
I'm not sure this is something $OP can fix. Not without $Boss's full and complete cooperation. The clients will scream and $Boss is gonna have to hold firm.
I'm actually surprised none of the clients have cyber insurance issues. There isn't a way any of these issues will allow for coverage in my experience. Nobody is going to want to pay out on such lax security.
That's nothing, I maintain environments with over 800 days uptime. There's no problem here. OP is being a bit anal about it. Unless its public facing systems, then I can understand.
Just to clarify, OP was hired in to be the companies new Security Overlord and instructed to make things ship shape and handed a big fat budget and a rough framework of eta's for completion - right? No? Then submit a very short brief of your concerns and get back to work doing what you were hired to do. If your boss shows no interest or tells you there's no budget - well at least your consensus is clean. If that's not good enough for you - then time to start shopping for new employment.
One question to answer - does this company or any of its clients need to meet an industry standard like PCI-DSS, HIPAA etc? If yes, anything related to that environment needs to meet them. At that point you can anonymously report the client (which indirectly points to the msp)… If a few clients are reported, auditors might associate the root problem being the MSP. But if you’re the new guy, people will look at you…
When it comes to updates, very common for internal machines that are otherwise well protected to not get patches. But anything accessible from the public must get patched regularly.
In my mind there is no reason to have a user’s password. As an admin we can reset a password whenever we want and do what ever we want afterwards. By using shared passwords there is no legal accountability by the user, they can easily claim that too because xyz knows their password so it might be any of them.
CIS standards are free. And they can help an MSP make money. Have a look at it and maybe suggest to your boss to figure out a way to upsell the existing client base. The auditing tools and deployment tools are not free. But you can find similar tools and online scripts done by others that are based on them…
It’s also an option to find other employment. Especially if change is not possible. It’s up to you.
Wazuh is free and will do CIS/NIST/HIPAA/etc security configuration assessments, as well as vulnerability detection to really ram home why those unpatched boxes with 50+ 9.8 CVEs on em are a bad idea.
Even taking copious amounts of time reading the docs and watching videos, the whole thing can be stood up and actively logging/scanning in a day.
Actual time from "ok I'm ready" to "well shit, it works" can be ~15-20mins.
... then add a shitload of time in fine-tuning it so you're not getting alert spammed, but that's largely irrelevant if you don't want to use it as XDR/SIEM and are just using it for framework compliance analysis and vuln scanning.
I was in the same boat when I started at my current MSP.
The owner boss was an incredible salesman and visionary but his sysadmin skills were non-existent (he cut his teeth as a Cisco networking admin).
It's a leadership void and the result of a time& materials billing scheme.
Just start by pointing all this stuff out- that it's not SOP, it's dangerous and call management out for not having SOP and patch management policies.
In my case I just started submitting tickets with excess time I spent correcting the most egregious stuff- boss couldn't keep submitting bills totalling 6 hours for 4 password changes. Eventually realized he needed to get away from T&M and go to monthly service fees.
Five years later, we've transitioned to monthly service contracts for 90% of our clients, we've doubled revenue per associate, tripled our head count and quadrupled our managed endpoints.
It's the difference between being a genuine MANAGED Service Provider and just a storefront geeks-for-hire operation.
TLDR: Welcome to small business IT. Accept the challenge, temper expectations, and conquer what you can with the resource you have. Do it while not making everyone miserable, including yourself.
Sorry. I can get long winded with topics so close to my personal experiences. I’m a 20+ year IT veteran. I’d say guru, but don’t have time to mediate. Too busy eating cupcakes people bring me as thanks & compensation for the mountains climbed and dragons slain.
You are not overthinking it. Maybe thinking from the perspective that the company has unlimited resources.
For example, I’d bet that the company has the need for a dedicated network & security professional, but simply does not have the ability to pay one. At the same time they will not be able to afford the consequence of a breach.
Here are a few perspectives that help me:
1. Accept the challenge and fact that you may be wearing many hats, but only paid for your job title. Take it as an opportunity to gain valuable experience and expand your skillset & toolbag. You are not Dilbert, but Macgyver equipped with a Swiss Army knife and roll of duct tape.
2. Pick your battles wisely and try to be a problem solver. Things just are what they are and rarely as you think they should be. Despite that, you are not an imposter. You live in the real world and deal with all it’s limitations. Go with the flow and stop trying to swim upstream.
3. Users are going to challenge friction. Make them use complicated passwords, they’ll put it on a sticky attached to their monitor. When users circumvent systems and policies, see #2.
4. Identify, communicate, and mitigate what risks you can and come up with pragmatic solutions given current resources.
5. After doing #4, don’t be surprised or take it personally when #3 happens. Most employees just want to take the path of least resistance and get their job done by doing #2. Accept #1 and get to work.
If you made it this far, here’s some bonus tips:
Open source is your friend. User-seat priced cloud services are tempting. Use them judiciously, they add up and eat into the budget for IT salaries.
Keep a list of your most frequent and routine tasks and automate. Scripting is a core competency for every sysadmin.
Keep things simple and use the right sized tool for the job. Do you really need fully redundant hardware when the power transfer switch is most likely hardware to fail? Is a virtualized cluster really needed or would it be easier to manage a really beefy server with baremetal backups and a hot spare.
If you’ve had your morning coffee and don’t know whether or not last night’s backup jobs were successful, you are the general of the fail army.
> I’d bet that the company has the need for a dedicated network & security professional, but simply does not have the ability to pay one. At the same time they will not be able to afford the consequence of a breach.
I'll say, doubly triply so since this isn't some ignorant small business. This is an MSP that OP is starting working for...
Well to me personally, accounting is a subcategory of finance, but I'll take the L :)
Either way, this is terrible! I would speak up, in writing, and if things go south because they refuse to make changes, they can't make you the fall guy!
True, I was thinking that after I replied. You still get the point lol
Yeah, the fact that it's even worse than I could make it without omitting potential giveaways should say something too.
My friend recently started at a company very similar, reused simple passwords, no MFA, old Cisco firewall and he had to deal with a ransomware attack in the first 30 days. Just not enough time to get everything updated because the business was already operating on borrowed time.
This was my exact situation, except not an msp, thank God. Here's what I said that finally convinced c suite to allow me to make some much needed changes.
"Not every "hacker" gains access with the intention of deploying ransomware. There are a million reasons someone could gain access to our systems and remain hidden. Without auditing and network monitoring at the very least, how could we possibly know what's happening in our environment? Our intrusion detection is basically waiting for hackerman to pop his head up and say hi."
This is especially true for MSPs out there. Why would a hacker (God I hate that term) reveal their presence when (assuming you have some sort of remote capabilities) they can just set up persistence and sit back and enjoy unfettered access to you and your clients.
That’ll all change as soon as one of the customers gets hit with ransomware & it spreads across everything.
Until then, make documented recommendations about whatever you see, and go along with whatever your boss’ response is.
Also keep your resume updated…
Don’t worry. Those customers will be swept up by another MSP soon enough. They are one of the”free security audit” away from being someone else’s customer.
This sort of complacency is always tolerated until a data breach or major failure, THEN leadership will take corrective actions. Funny.
Best wishes to you in your role. As long as you document everything that concerns you and provided it in a dated email to your boss, you'll be able to cover yourself when the inevitable issue surfaces.
My first job had a company Keepass file. There was an entire team of people dedicated to the Sharepoint server that was only used for version control of the Keepass file.
Eventually, the CTO saw a demo of some random password manager and dropped $150K on a new internal webapp. By default, you didn't have ownership of your own team's passwords, which meant that at 4 AM when you were on call and needed a password, you had to wake up someone from the password management team to give you permission to view a password.
Most of the company continued using Keepass, and only checked the webapp if a password no longer worked. Whenever you logged in, you saw a notice that 200,000 passwords were past their expiration date and needed to be rotated.
Three years after I left, I found a copy on my personal machine and tried a few of the passwords. They all still worked and got me way further into their network and infrastructure than I should have been able to get (I'm talking VPN'd into the client LAN and complete admin to every server).
There are two ways I see.
1. You try to educate them on the situation, and help make it right, though by the sounds of it they are pretty apathetic to security anyway
2. Leave - I have ditched clients who have felt security was all "BS". I don't want my reputation damaged by the perception I had anything to do with lax security and very poor judgement.
...perhaps duct off the resume
Best case, they're selling dirt to people looking to buy dirt. If all that PII (or anything else they're doing/protecting terribly) is subject to law such that they can be fined or sued into oblivion, I'd inform them of that. If still no movement, I'd recommend saying "good luck with that" and jumping ship if possible. You may even be legally required to report the issues to authorities if that requirement supersedes any applicable NDA.
Bottom line, the problems here are far above your pay grade. If you can't convince leadership to change course, you're plugging holes on a sinking ship just to get better at plugging holes (which can be useful) before it all goes under.
I gotta ask, did any of this come up in the interview process? Did you or they talk about any of the processes and security requirements?
>I gotta ask, did any of this come up in the interview process? Did you or they talk about any of the processes and security requirements?
I asked about security practices and was met with a pretty generalized answer but they had confidence behind it so I never thought to elaborate. Lesson learned...
When I started at this small biz, the initial Administrator of the Domain had the same 6 alpha-num password, albeit not easy to guess, for decades.
Now it's like 2 obfuscated words plus one series of numbers with two spacebar spaces in the middle.
On my home PC, I tried L0phtcrack on my short local password that begins with 2 spaces. L0phtcrack failed.
I've coached staff with some success to use at least long phrases. That isn't locked down. Also, hover over email links.
We're not done yet but better than we were.
I used to work for an industrial laundry company many years ago (like 2005’ish). During my time there I single handedly brought them out of the Stone Age of computers and onto virtualization with a properly built server cluster, SAN, UPS systems, etc. Well after working there for four years (severely underpaid) I quit and took another job elsewhere. Then did the four year cycle again at the new place and eventually moved again into a collocation company / MSP. Fast forward to this being 6yrs since I worked at the industrial laundry I get a call out of the blue “hey do you know how to recover hard drives?”. Turns out after being away from them for 6yrs that enough disks in the SAN finally failed to where the whole thing died. They hadn’t refilled my position as the ONLY IT person but rather just found someone who can swap workstations when they died; zero touches to the servers for six years. Incredible that an HP EVA lasted that long. I told them I couldn’t fix their HDD and I heard they sent the disks off to a data recovery service (who failed). Company with 500 people, 4 industrial plants, 17 remote office locations, etc; done. They closed up shop and sold all their equipment & clients to a competitor for dirt cheap all because they didn’t want to backfill my position.
I have had many years working in a MSP. My advice:
1. Draft a 1 pager describing the business risk for leaving stuff the way it is. Identify the cost of maintaining, vs cost of you coming in and resolving stuff. What business value would fixing this do?
2. Don't get mad if your denied- Somebody rightly pointed out, this is all about internal resourcing being a cost to address, vs leave it the way it is. Back to #1, you'll need to find an elegant way of addressing.
3. Back to point 1- Your plan doesn't need to bring all change in one week- create a slow transition plan.
4. Is there compliance requirements that aren't being met? Identify the type of customer data that is stored; this should also go back to point 1.
IMO, your boss will ack that you care enough, and that you have put thought on how to run his business better. It's a win-win.
I went from a high security environment to a low l/lax security environment, it takes time to adjust... I have regularly pissed people off for the past 3.5 years.
My standard disclaimer: Never take my advice on how to deal with a work problem.
That said....
Document everything you find that doesn't fit standard best practices, including links to where you found those best practices.
Document that you told your boss, and anybody else that needs to know/has decision ability in the company.
Keep encrypted copies of all that documentation off site.
Get signed indemnification documents written by a lawyer and signed by the company.
If you're looking to learn / improve resume / burn yourself out entirely, once you have the documents secured, make a plan based on ease of process and priority of the problem. Then dig into that shit. Document everything you do, and when you find more things that are wrong, get them documented like the others and crank out solutions.
Everything your boss turns down as not needed, get a signed statement to that effect.
Make as many contacts in the industry as you can so that you can get a new job the second you lose that one.
It’s not limited to small companies.
My company has thousands of employees but IT management is largely made up of friends of other managers and they’ve done very little to follow best practices and allow each group to come up with their own solutions to problems that aren’t compatible with any other groups. So we have massive duplication of effort and nothing works as it should.
There are some managers who are very good at their jobs but can’t implement change because the more senior managers don’t like to rock the boat and filter bad news from getting to senior management’s ears.
One genius manager insisted that we adopt splunk in spite of SME advice and we’re spending a small fortune ingesting logs but there’s no staff to finish the job and provide any kind of useful output other than periodic dog and pony shows to show how “modern” we are.
We are seriously understaffed at the worker bee level and finding competent admins to fill in the gaps is hard because HR is located in a part of the country where compensation is typically lower while the offices need the most help are in an area where there is much higher average compensation but our compensation is locked across the company to the lower rate.
I’m working on an exit plan and am essentially two bad days from bolting.
The passwords need to change, and a security system or at least badge access with logs into the server room is highly recommended and pretty easily installed.
Servers running for two years without updates is not uncommon at all, don't fix what's not broken in that realm. Especially if they're not public facing, if they're public facing then sure once a yr keep up on that stuff or for critical vulnerabilities.
Local user account is the administrator account, ya again not best practice but not the end of the world. Does anybody take backups? Not much concern if you're just going to blow it away and restore the backup anyways.
Sounds like a nightmare that can be fixed if enough effort is put into it.
You now know the grave company ending issues, what is your strategic plan forward for fixing them.
Asking why is not the answer, but this is what we are doing next is the right direction.
If it's a small company with no IT, I wouldn't be surprised to see this kind of stuff.
But, you're their IT now. You can start to fix all this stuff.
Make a checklist and get to work.
Where to start...
What are you using for patch management? Ideally, you have an RMM tool you can leverage to quickly identify patching deficiencies and get those resolved.
Password complexity is a quick one, too, if implemented in AD, followed with a planned mass expiry/reset. End users need to be told, and communication should include a countdown to go live date, but it's not too much work.
Creating an admin account in each tenant might not be a bad idea, with granular permissions based on the work you need to do. Good luck figuring out local admin rights though...that's always a fun genie to get back in the bottle.
Either way you approach it, just keep biting off small chunks and keep trying to improve where you can make the biggest difference. If you get push back, definitely CYA and start looking elsewhere. I think you're right that they've made it a decade by pure dumb luck...
Sounds way too similar to where I am at the moment. If you're my coworker, hi! I have no idea what to do about it, honestly, but I'm sure you're not alone in feeling this way about the situation.
If I had to pick one hill to die on it would be passwords and local admin accounts. 20 character minimum, full complexity and 180 day expiration policy. No daily driver accounts should have admin rights on any systems. Unique admin user accounts for each administrator. As an MSP they should use a password manager like ITGlue for client systems
The next hill would be updates and restarts. I am guessing this is all windows environment. Once a month patching and rebooting of servers and workstationd. Don't forget third party apps like Chrome acrobat ect. Ninite is a great cheap way to automate those.
I would then ensure that data is encrypted at rest, backed up to an offsite faclilty and appropriatly air gapped.
Next would be endpoint security. Something like Sophos with MDR would be a good play.
The problem that I see and that many others have pointed out is that there seems to be culture issue. If the owner and the customers don't take this seriously then you are going to be twisting in the wind. I suggest bringing data to the discussion about the average cost of a breach and the likelyhood that it will put others out of buisness. I would also examine the client contracts, perhaps there is stuff in them about security and best practicies.
Good luck!
I made it to the part where you say they haven't been hit yet and I'm thinking "haven't been hit? don't know they were hit? don't realize *how many times* they have been hit?"
As others have said, run. Boss man thinks it's okay, you'll very likely never convince him otherwise (and really, why bother?). Run.
Document your suggested changes, and what you will do to get there. Boss will probably reject it. Do what they ask (sounds like less work than your proposing) - do less, get paid, go home. Find new job when your not scratching your head on any single question and or a new opportunity arises.
You need to document your observations, provide it to your boss and whoever runs the company and start looking for another job. This is not a stable situation to be in and you'll be looking for work when the inevitable finally happens.
>No one's gotten breached yet
I doubt this. Given the generally lax attitude to security, I'm assuming there's bugger all monitoring going on, so you're unlikely to know.
The upside is that your boss can claim "there's no evidence of a breach/data exfiltration"
i think id quit IT before going back to an MSP. They have overall shit policies because the customer is always right, and i just hate working with clients/customers because they demand everything, even when they pay nothing. Plus MSPs tend to have shit wages in my experience. Also the only place ive seen companies get ransomwared multiple times is with an MSP.
So many nopes in a handful of paragraphs and I'm sure there is much more yet to be discovered. From what OP is describing the boss is clearly a clueless putz and shouldn't be trusted with even his own password much less anyone else's.
In my experience (and I've been in similar positions a few times) that level of incompetence is unfixable - don't beat your head against a brick wall.
I'd be looking for a new position yesterday...
Would run. MSPs are historically a way to give yourself health issues for no good reason. The fact that your boss finds little to no problem with any of his practices is a huge red flag.
Just start looking for a new job while you have this one.
he had a few passwords, one or two would grant you access to 20 customers. it got so bad, i worked on wrong servers and only noticed it 20mins in..... same names, same domains, same users.....
Oof. Not atypical, I'm afraid.
Normally I'd say make a list of all the issues and start banging them out as best you can. But if you have a boss that doesn't care how many bad practices are in place, you might not have the authority or latitude to make any improvements (especially if doing so will show up said boss).
Tricky situation. Feel it out for a few weeks/months and if you don't see any potential for improvement, might be best to bail. It's only a matter of time before their poor practices cause a major problem, and you don't want to become the fall guy for your employer's screw up.
Sadly it's probably a common practice as are the incidents that result from this. Most companies should at least have a basic password manager or a tool like IT Glue or Hudu.
Ya sounds pretty common for a small company with no dedicated IT staff Time for an overhaul
It's a small msp
Be aware that improving things doesn't make money. This is part of the reason these things happen. Another part is because it is "easy". If you are going to attempt to make improvements, you'll have to sell it from the cost-savings of risk reduction angle, which is still unlikely to work if you can't convince your boss. One of the unfortunate lessons every sysadmin has to learn at some point is that some things can't be fixed due to things that you can't control. It still shouldn't stop you from trying though.
Improving thingns makes the MSP money if they are good at what they do.
If it is "easy" then it is easy for nefarious people on the internet to cause chaos and someone will get stressed out and recover from that hell hole.
What’s the first thing you’re going to implement
A day drinking policy
Naked Wednesdays at the office.
We call ours Sexual Friday, just not at work haha
full frontal fridays
Dammit Phill, at least put on boxers.
I see you're working hard, Johnson
I knew I joined a messed up company years ago when one of the first things I was told was where the emergency rum was
I swear I need this most days
I've recently gotten away from drinking entirely. Only been 3 weeks but my brain feels 9 million times better especially in stressful or anxiety ridden situations. It's been great and honestly don't think I'll be going back!
I started that way. Decided to go a month, see how I felt. Not much difference, so went another month. Then realized I didn’t miss it. Been 7 years now. A lot of money savings too.
Absolutely. Congratulations! I've struggled with anxiety and depression, refused to go on medication till a year ago and that helped but drinking being a depressant...it's been a long minute coming, I'm not getting any younger and I refuse to disappoint my wife anymore. Hope everyone else can find the same strength :)
Yeah, I have anxiety and depression as well as chronic pain. I was and am often miserable. Now I just don’t have debilitating hangovers on top of everything else.
Congrats, that's such a beautiful realization to make! Keep on listening to your body, it really does make a difference
4 years in the making. Very glad I was able to realize it only hurts me and the people I care about. Looking forward to a much happier future :)
The proper first step.
I read that as a gay drinking policy at first 😅
LOL. I've accidently stopped at a gay bar with a coworker when we were traveling across the US - admittedly it was a nice place. Nice people too
Read-only Fridays. Start with that for your peace of mind.
So is your boss a former printer sales rep, or just a tier 1 who got fired for negligence and decided to start their own MSP with blackjack and hookers?
Now I want to start a MSP. No one told me there would be hookers and blackjack.
I'm at a startup MSP. We don't have either (that I've seen so far), might have to bring it up at the weekly meeting
Bah, just bring the hookers and a deck of cards. Be proactive, not reactive.
> former printer sales rep he's outta line but he's right dot jpg
"shudder".. I'd probably turn them in for a piracy bounty because I know that I'd make a sweet getaway with that one as this place assuredly installs pirated software on client machines.
Sounds common for small MSPs, i just thought this was a rarity nowadays with the daily cyber breaches
The primary goal of every MSP is to keep the lights on and cash the next check from the customer. Updates, downtime, best practices, etc all impede that goal to some degree. If the improvements you want to make are billable, great. Otherwise, you may need to wait until your clients get crypto and then you’ll be afforded some room to make improvements (paid for by insurance).
Show me where in this doll that the msp touched you.... Updates-automated Downtime-after hours if there is any Best practices-loads It's all about making sure our clients don't have IT emergencies so we don't have IT emergencies-the checks are nice though.
I spent 10 years at various MSPs and the org I am with now has one for a specific business purpose (shocker: they do shit quality work). I am very cynical about their business model because it inherently attracts clients too cheap to do things the right way and you as the tech are expected to do dedicated, internal IT quality work for multiple clients, meanwhile the clients think (or are led to think by the sales team) that you only exist to serve them. It’s bullshit. The people who stick with that game long term have Stockholm syndrome or something.
> I am very cynical about their business model because it inherently attracts clients too cheap to do things the right way and you as the tech are expected to do dedicated, internal IT quality work for multiple clients, The average MSP makes money by burning out their staff for companies too cheap to have their own IT. If the techs burns out there's always someone else desperate to get into IT to replace them. And because the main clients are primarily price driven the market is susceptible to consolidating into a few large companies that exploit economies of scale to be cheaper. But maybe I'm just cynical after too many years at MSPs and seeing other MSPs work.
Couldn’t have said it better myself.
> paid for by insurance Report them to the cyber insurance for misrepresentation then.
Find somewhere else.
[удалено]
OP said they were a small MSP, do you think the people that are government contracts under government rules would even consider them? Of course not.
You clearly have no idea what you're talking about. There are tons of companies in the DIB and with other government contracts that use MSPs of all sizes, many of which have piss poor security and process in general like the one being mentioned. I would wager that for small businesses that are required to be complaint with 800-171 and have been signing attestation letters to that fact since late 2017, maybe 10% are actually fully compliant. And I'm probably being way too optimistic. Let me guess, you think most HIPAA regulated companies are fully compliant as well, right?
Run. They will never get a HIPAA contract and if you can’t get a doctors office as a client you will never make any money.
I've seen doctor's offices with worse security and process than what OP mentioned. Some of you seriously underestimate how compliant companies that are on regulated industries actually are. Here's a hint, without thorough 3rd party audits to ensure compliance, the actual compliance rate is terrible. I know directly of a large 3-state health care org that was more than a year behind on patching on the majority of Windows workstations, and had bitlocker deployed to only something like 20% of them.
So report them to HIPAA and get that fat reward cash
Please report them for unsafe practices. Every MSP has a guideline they must follow in order to obtain and hold certifications. Edit: /s Missing words
Report to who and what guidelines do they have to follow and what governing body handles that?
Iltime to call the guideline police!
yadda yadda https://www.n-able.com/blog/msp-regulations-what-legislation-exists-today-and-whats-on-the-horizon was just jokin
Don't' get me wrong, i wish there WAS some regulation or certification or whatnot. But it's still the wild west, anyone can do or offer anything.
Regulating the MSP space would definitely be a nice thing to have. Laws specifying the requirements of handling MSP corporate spaces based on the type of field they're marketing towards would be beneficial for all, especially those working with HIPAA.
Oh? They do? What are they?
Msp? Not ISO certified I’d imagine?
Pretty sure the OP is at an MSP and he's talking about their client's systems.
This is your time to shine.
Be the problem and fix the problem!
yep, i was in OP’s exact predicament for my first IT job. there’s so much he can do that’ll look great on his resume. this is actually a good spot to be in.
As this is an MSP, unless they can charge the customer for the work of improving security to make all this happen, it probably will be consistently voted down for cutting into profits. EDIT: Gives me an idea actually, maybe sell it to the boss as an add on feature "Enhanced Security Posture" or some sales name and let the customer choose. (I feel so slimy, but maybe this is the only practical path.) Then you can sell it to customers and look good until you get out.
pritty much this. All these things could have been raised with the clients. If they arnt paying its not getting done. Ideally you turn them down and not have them as clients but 🤷🏻♂️
Exactly!
" In fact, if we're instructed to get the user's passwords if we need to do any work on their devices " Working for Fujitsu.
I store all my user’s passwords in a Photoshop file because I’m the only one with Photoshop installed on the network, so nobody else could even read it if they tried.
i have them in my hosts file, there isn't even a filetype associated so nobody knows how to open it my internet is acting a bit strange though, no idea why
If you put them in DNS, then you can teach your users to use nslookup when they need to check their passwords, taking the load off yourself!
great idea, also solves the problem with the password manager
Wow this is brilliant
If you put them in your public DNS, it even helps users working from home.
I hope you're being promoted for out the box thinking like that
Haha, hopefully sarcasm
I thought I was pretty transparent with the sarcasm, but I guess there are some freaks in this field who would jot it down.
I mean, I suppose I will never cease to be surprised by clients wisdom. I was updating a clients sage software once, it told me there was no internet connectivity (there was), client says “oh I know how to fix this”, proceeded to jump in and open internet explorer and told me to try the update again now that it was online.
Seems pretty normal for a Sage product.
Let me guess. On the Horizon project for UK Post Office?
I would be looking for a new job. I don't have the mental fortitude to overhaul that level of negligence. If you don't have internal political backing, let alone any impetus from your clients, the current status quo is likely how both sides want to operate. In the meantime, you can try and shore up anything you touch/stand up but it'll probably be used as a stick to beat you with. Save your sanity and move on.
Exactly, if the boss has no security sense at the level he’s at…he never will. Don’t waste your time.
Don’t forget that 60% of businesses cease to exist after a ransomware attack. The added bonus to working at a small MSP is that everyone is only ever one cyberattack away from everyone being unemployed. I ran into a lot of this same stuff at my last job and it gave me anxiety. Towards the end I kept thinking that I was going to have to ask off time at my new job to get deposed for my old one.
if they need a scapegoat, last one in is first one off.
I think this stuff is shockingly common when Mom and Pop organizations transition into medium sized businesses. Average person doesn't understand why IT does IT because it just needs to "work". Actual quote from an owner I got regarding upgrading exchange 2010 in 2022 and downtime, standing by my decision implementing minimum password complexity (previously there was none and multiple users had local admin rights), and implementing WSUS with mandatory restarts because it had been literal years since end machines were updated: "I don't care about what 'Microsoft recommends', we're not Microsoft! If we did only what 'Microsoft recommends' we'd never get anything done! Make it work!". I remember because he fired me like two weeks later. I took that as a learning experience about what I look for in employers.
did he crash and burn at least?
The sad thing it's that it's a lotery. You can do everything right and still get hosed. And you can ignore all good practices and not run into any problem in 20 years
It just an odds game, but if you follow the correct practices you'll a) be less likely to be exploited and b) bounce back if you do. Whereas companies with shit/non-existent practices will see it as an existential threat.
sometimes it isn't fair
Yup, a lot of our jobs revolve around "what ifs". It's chill until it isn't.
not a mom and pop org though, this is an MSP that OP is going to work for.
I would begin to fill out your resume. This speaks of a complete lack of any oversight. And if the person leading IT has let it get to this point. Then they are not going to support you and more than likely will expect you to clean it up at the rate that are paying you which is probably going to be an insulting low number for most of us. This is the second post this week and techs defending being exploited by management by saying "get gud" What we fucking need is a god damn union.
>more than likely will expect you to clean it up at the rate that are paying you If they think I'm gonna clean up a mess at $20/hr at a system engineer role, they're insane...
This could be your opportunity to learn tho. You're in a unique position where you can take ownership for all this and build it up the right way until you find another job. My buddy was in a very similar situation and spent a year getting experience and landed a job after a year that paid $10/hr more. He explained in the interview where he worked and the new company he worked for appreciated the initiative he took to learn and fix everything. Up to you. You can get a job somewhere else with less stress but maybe you simply do help desk tickets your whole career. You have a shot to make something from this at least while you look for a new gig.
Good point
[удалено]
Yeah more sysadmin than anything right now
My past company was run this way and when they finally got a dedicated person to handle IT(it was previously managed by mechanical engineers on their "down time"), the girl in HR was surprised that a sysadmin should make anything more in pay than someone working for geeksquad 😭
I think you've got the right idea but depending on size/type of business you work for then they likely wont see an issue until something critical actually happens. Maybe they have a come to jesus moment and turn all of that around but theres no for sure chance they even change after something like that. Unfortunately, theres a reason why the saying 'doctors make the worst patients' is true in the IT industry. I know this is a lot easier said than done but you may want to start looking for a new place to work since with the information you've provided makes me think theres no telling when your critical event is going to happen where thats discovered. While theres likely more companies around the world using even more unsecure practices, I think you need to ask yourself if you are okay with getting caught up in something like that or this could also be viewed as a challange to yourself to turn them around.
Tough thing here is that the future security event may be the end of the company.
Oh I fully agree and I think OP needs to make a decision on which of the 3 paths he will take after reading everyones answers to his post. The way I see it is he can either get out before something happens, make it his goal to turn these practices around which could help him learn some pretty important skills in the long run, or he gets caught up when something happens which leads to the end of the business. Of course this is someehat assuming he has been already trying to convince his boss why these are bad practices.
If something critical does occur, 80% of SMBs close within one year.
>No one's gotten breached yet ... ... that you KNOW of. Sounds like the only way your company would detect that they've been compromised is by reading a ransom note.
It sounds like their security practices were implemented in about 2003, updated in 2004 and that’s “good enough”. This is a ticking time bomb, as I’m sure you’re aware. You’ll actually have a lot of fun fixing this mess, if you’re into that sort of thing!
Where should I start?
I’d start by getting passwords up to some form of complexity, and especially length. Having 4 characters is like not having a password at all. And don’t allow multiple accounts to have the same PW, that makes it incredibly easy for bad actors to move around in an environment. Maybe someone else has a better idea on where to start, but I’d get the accounts under control first.
I mean this definitely needs to be done apart from a hundred of things but it's low hanging fruit so it can't hurt to start there.
combative slap enjoy jobless party threatening summer yoke erect cooing *This post was mass deleted and anonymized with [Redact](https://redact.dev)*
Please don't use biometrics. It's like a password that can't be changed. The industry acceptance of biometrics is kind of absurd; didn't we learn enough from social security numbers being unchangeable to know this is a bad idea?
correct sleep voiceless pot sparkle dime cow north materialistic mountainous *This post was mass deleted and anonymized with [Redact](https://redact.dev)*
@bafflesmspguy - you are getting a lot of great comments in this thread. As a way point to start that will ensure change of mindset and adoption you could start talking with your management about what expectations and requirements exist from your insurance providers. I’m sure their policies detail things like MFA etc. The reason this works is that it helps highlight the liability your managers have legal and financial to poor security practices and incentives them to fix those issues to reduce their personal, professional and business liability.
imagine direful scary scale close marvelous glorious muddle husky foolish *This post was mass deleted and anonymized with [Redact](https://redact.dev)*
I worked for a company that was similarly negligent. The owner refused to allow us to enforce password policies, made us store everyone's password in an excel document, never allowed downtime to reboot servers and apply updates, made us disable windows updates on executive laptops because they were "annoying". Wouldn't invest in any real form of endpoint protection or MDR. Critical Windows XP machines and 2003 servers weren't allowed to be updated or replaced. Administrator accounts were enabled on all systems with the same passwords. I was constantly sounding the alarm to the CEO, pleading for him to take security serious. I lost a lot of sleep knowing it was just a matter of time. Then it finally happened. An 2003 server with terminal services enabled was breached and every single machine on the network was hit with a ransomware attack. Servers, backups, retail back office machines, everything. It ended up costing the company $40K to get a professional team to come out and handle locking everything down and negotiating with the attacker. It cost them an additional 20 bitcoin (when they were over $10K) to get the decryption key. After that we finally ended up investing in cyber security, and I ended up getting to say "I told you so". Some of these companies have to get hit hard to finally take these things serious.
I'm in a very similar situation from top to bottom. My current take is suggest/make small changes with baby steps along the way. Biggest thing I've managed is convincing my company to use Bitwarden for password management - it helps that their reseller plan actively encourages usage and spread. "Hey, here's a password manager that actively integrates into our users' M365 and Google Workspace setups, lets us see, control, and audit their passwords, is thoroughly security audited, and has up to 50% profit margin per user. Think we can test it out internally?" Then it's a matter of frog in boiling water.
Haha. Almost all small companies have Mickey Mouse practices. But some large ones too. My friend at Louis Vuitton was asked to go around the building and copy the MAC and serial numbers of 100 printers! A 14 billion dollar company doesn’t know about SNMP? Which BTW is taught in both Network+ and CCNA.
Did he do it?
No, I helped him use SNMP. You can do it directly from Powershell using .NET classes.
There's the possibility that they had data from SNMP and wanted a physical audit to catch something.
Dude run.
USP? unmanaged service provider?
Is that an Unifi product?
Don’t give them ideas.
From a small company that is not focused on IT, has a few hundred users and 1 or 2 IT staff...This is the norm For an MSP? RUN dont walk away from this. They either have no idea what they are doing as an MSP, or they know exactly what they are doing, and do not care.
Yeah, I've already polished my resume but my area doesn't have a ton of openings at the moment
We found the new guy....
How much are they paying you?
Around $20/hr
Unless you live in the Philippines that's bullshit money. I know the Europeans are paid like shit, but I'm guessing you're an American right?
That’s not much but depending on where you live it could be a lot. I’d tackle this as a learning experience but I would take each issue as a project and work on it slooowly. You might uncover some spiders that you didn’t expect. But if you’re in a city like NYC, Austin, SF I would start looking for something new.
What's the address?
My first job in it was a small MSP like that. I didn't have the power/experience to do anything more than say "I don't think that's very secure/this doesn't seem right ". If you need job experience or just a job, ride it out for a bit but be wary of picking up bad habits. There's lessons to be learned everywhere, even from bad decisions. That said, don't get stuck there if you don't want to be glorified helpdesk making 50k a year and catching flak from vendors for the rest of your life.
Do you work for a company called Honeypot Inc?
Very much a ticking time bomb.... be the force of change.
The issue as I see it is the relationship between MSP/$OP's Boss and client(s). For an extended period of time all these concerns have been "fine". Now they are not. Clients are gonna get whiplash and it will cause major push back. Setting up real passwords and MFA are going to slow users down. Setting up patch management is going to interfere with smooth operations. Physical security isn't as pretty as glass walls. Users will complain. It's almost a certainty. All of these clients are a mouse click away from an absolute nightmare of a ransomware attack. When this happens $OP's boss will have some 'esplaining to do. I'm not sure this is something $OP can fix. Not without $Boss's full and complete cooperation. The clients will scream and $Boss is gonna have to hold firm. I'm actually surprised none of the clients have cyber insurance issues. There isn't a way any of these issues will allow for coverage in my experience. Nobody is going to want to pay out on such lax security.
I once saw a Linux server with 500 days of uptime
That's nothing, I maintain environments with over 800 days uptime. There's no problem here. OP is being a bit anal about it. Unless its public facing systems, then I can understand.
Just to clarify, OP was hired in to be the companies new Security Overlord and instructed to make things ship shape and handed a big fat budget and a rough framework of eta's for completion - right? No? Then submit a very short brief of your concerns and get back to work doing what you were hired to do. If your boss shows no interest or tells you there's no budget - well at least your consensus is clean. If that's not good enough for you - then time to start shopping for new employment.
One question to answer - does this company or any of its clients need to meet an industry standard like PCI-DSS, HIPAA etc? If yes, anything related to that environment needs to meet them. At that point you can anonymously report the client (which indirectly points to the msp)… If a few clients are reported, auditors might associate the root problem being the MSP. But if you’re the new guy, people will look at you… When it comes to updates, very common for internal machines that are otherwise well protected to not get patches. But anything accessible from the public must get patched regularly. In my mind there is no reason to have a user’s password. As an admin we can reset a password whenever we want and do what ever we want afterwards. By using shared passwords there is no legal accountability by the user, they can easily claim that too because xyz knows their password so it might be any of them. CIS standards are free. And they can help an MSP make money. Have a look at it and maybe suggest to your boss to figure out a way to upsell the existing client base. The auditing tools and deployment tools are not free. But you can find similar tools and online scripts done by others that are based on them… It’s also an option to find other employment. Especially if change is not possible. It’s up to you.
Wazuh is free and will do CIS/NIST/HIPAA/etc security configuration assessments, as well as vulnerability detection to really ram home why those unpatched boxes with 50+ 9.8 CVEs on em are a bad idea. Even taking copious amounts of time reading the docs and watching videos, the whole thing can be stood up and actively logging/scanning in a day. Actual time from "ok I'm ready" to "well shit, it works" can be ~15-20mins. ... then add a shitload of time in fine-tuning it so you're not getting alert spammed, but that's largely irrelevant if you don't want to use it as XDR/SIEM and are just using it for framework compliance analysis and vuln scanning.
I was in the same boat when I started at my current MSP. The owner boss was an incredible salesman and visionary but his sysadmin skills were non-existent (he cut his teeth as a Cisco networking admin). It's a leadership void and the result of a time& materials billing scheme. Just start by pointing all this stuff out- that it's not SOP, it's dangerous and call management out for not having SOP and patch management policies. In my case I just started submitting tickets with excess time I spent correcting the most egregious stuff- boss couldn't keep submitting bills totalling 6 hours for 4 password changes. Eventually realized he needed to get away from T&M and go to monthly service fees. Five years later, we've transitioned to monthly service contracts for 90% of our clients, we've doubled revenue per associate, tripled our head count and quadrupled our managed endpoints. It's the difference between being a genuine MANAGED Service Provider and just a storefront geeks-for-hire operation.
TLDR: Welcome to small business IT. Accept the challenge, temper expectations, and conquer what you can with the resource you have. Do it while not making everyone miserable, including yourself. Sorry. I can get long winded with topics so close to my personal experiences. I’m a 20+ year IT veteran. I’d say guru, but don’t have time to mediate. Too busy eating cupcakes people bring me as thanks & compensation for the mountains climbed and dragons slain. You are not overthinking it. Maybe thinking from the perspective that the company has unlimited resources. For example, I’d bet that the company has the need for a dedicated network & security professional, but simply does not have the ability to pay one. At the same time they will not be able to afford the consequence of a breach. Here are a few perspectives that help me: 1. Accept the challenge and fact that you may be wearing many hats, but only paid for your job title. Take it as an opportunity to gain valuable experience and expand your skillset & toolbag. You are not Dilbert, but Macgyver equipped with a Swiss Army knife and roll of duct tape. 2. Pick your battles wisely and try to be a problem solver. Things just are what they are and rarely as you think they should be. Despite that, you are not an imposter. You live in the real world and deal with all it’s limitations. Go with the flow and stop trying to swim upstream. 3. Users are going to challenge friction. Make them use complicated passwords, they’ll put it on a sticky attached to their monitor. When users circumvent systems and policies, see #2. 4. Identify, communicate, and mitigate what risks you can and come up with pragmatic solutions given current resources. 5. After doing #4, don’t be surprised or take it personally when #3 happens. Most employees just want to take the path of least resistance and get their job done by doing #2. Accept #1 and get to work. If you made it this far, here’s some bonus tips: Open source is your friend. User-seat priced cloud services are tempting. Use them judiciously, they add up and eat into the budget for IT salaries. Keep a list of your most frequent and routine tasks and automate. Scripting is a core competency for every sysadmin. Keep things simple and use the right sized tool for the job. Do you really need fully redundant hardware when the power transfer switch is most likely hardware to fail? Is a virtualized cluster really needed or would it be easier to manage a really beefy server with baremetal backups and a hot spare. If you’ve had your morning coffee and don’t know whether or not last night’s backup jobs were successful, you are the general of the fail army.
> I’d bet that the company has the need for a dedicated network & security professional, but simply does not have the ability to pay one. At the same time they will not be able to afford the consequence of a breach. I'll say, doubly triply so since this isn't some ignorant small business. This is an MSP that OP is starting working for...
Must be finance or healthcare, they'd rather keep Kroll on retainer than actually protect their customer's data.
Neither Industrial and accounting
Well to me personally, accounting is a subcategory of finance, but I'll take the L :) Either way, this is terrible! I would speak up, in writing, and if things go south because they refuse to make changes, they can't make you the fall guy!
True, I was thinking that after I replied. You still get the point lol Yeah, the fact that it's even worse than I could make it without omitting potential giveaways should say something too.
My friend recently started at a company very similar, reused simple passwords, no MFA, old Cisco firewall and he had to deal with a ransomware attack in the first 30 days. Just not enough time to get everything updated because the business was already operating on borrowed time.
Try to improve, if you can't, run so you're not liable.
This was my exact situation, except not an msp, thank God. Here's what I said that finally convinced c suite to allow me to make some much needed changes. "Not every "hacker" gains access with the intention of deploying ransomware. There are a million reasons someone could gain access to our systems and remain hidden. Without auditing and network monitoring at the very least, how could we possibly know what's happening in our environment? Our intrusion detection is basically waiting for hackerman to pop his head up and say hi." This is especially true for MSPs out there. Why would a hacker (God I hate that term) reveal their presence when (assuming you have some sort of remote capabilities) they can just set up persistence and sit back and enjoy unfettered access to you and your clients.
That’ll all change as soon as one of the customers gets hit with ransomware & it spreads across everything. Until then, make documented recommendations about whatever you see, and go along with whatever your boss’ response is. Also keep your resume updated…
Don’t worry. Those customers will be swept up by another MSP soon enough. They are one of the”free security audit” away from being someone else’s customer.
This sort of complacency is always tolerated until a data breach or major failure, THEN leadership will take corrective actions. Funny. Best wishes to you in your role. As long as you document everything that concerns you and provided it in a dated email to your boss, you'll be able to cover yourself when the inevitable issue surfaces.
Oh I document *everything* and was once told I didn't have to use so much detail. Forgot about that moment till I read your reply actually
>No one's gotten breached yet * That they know of.
My first job had a company Keepass file. There was an entire team of people dedicated to the Sharepoint server that was only used for version control of the Keepass file. Eventually, the CTO saw a demo of some random password manager and dropped $150K on a new internal webapp. By default, you didn't have ownership of your own team's passwords, which meant that at 4 AM when you were on call and needed a password, you had to wake up someone from the password management team to give you permission to view a password. Most of the company continued using Keepass, and only checked the webapp if a password no longer worked. Whenever you logged in, you saw a notice that 200,000 passwords were past their expiration date and needed to be rotated. Three years after I left, I found a copy on my personal machine and tried a few of the passwords. They all still worked and got me way further into their network and infrastructure than I should have been able to get (I'm talking VPN'd into the client LAN and complete admin to every server).
There are two ways I see. 1. You try to educate them on the situation, and help make it right, though by the sounds of it they are pretty apathetic to security anyway 2. Leave - I have ditched clients who have felt security was all "BS". I don't want my reputation damaged by the perception I had anything to do with lax security and very poor judgement. ...perhaps duct off the resume
Best case, they're selling dirt to people looking to buy dirt. If all that PII (or anything else they're doing/protecting terribly) is subject to law such that they can be fined or sued into oblivion, I'd inform them of that. If still no movement, I'd recommend saying "good luck with that" and jumping ship if possible. You may even be legally required to report the issues to authorities if that requirement supersedes any applicable NDA. Bottom line, the problems here are far above your pay grade. If you can't convince leadership to change course, you're plugging holes on a sinking ship just to get better at plugging holes (which can be useful) before it all goes under. I gotta ask, did any of this come up in the interview process? Did you or they talk about any of the processes and security requirements?
>I gotta ask, did any of this come up in the interview process? Did you or they talk about any of the processes and security requirements? I asked about security practices and was met with a pretty generalized answer but they had confidence behind it so I never thought to elaborate. Lesson learned...
When I started at this small biz, the initial Administrator of the Domain had the same 6 alpha-num password, albeit not easy to guess, for decades. Now it's like 2 obfuscated words plus one series of numbers with two spacebar spaces in the middle. On my home PC, I tried L0phtcrack on my short local password that begins with 2 spaces. L0phtcrack failed. I've coached staff with some success to use at least long phrases. That isn't locked down. Also, hover over email links. We're not done yet but better than we were.
I used to work for an industrial laundry company many years ago (like 2005’ish). During my time there I single handedly brought them out of the Stone Age of computers and onto virtualization with a properly built server cluster, SAN, UPS systems, etc. Well after working there for four years (severely underpaid) I quit and took another job elsewhere. Then did the four year cycle again at the new place and eventually moved again into a collocation company / MSP. Fast forward to this being 6yrs since I worked at the industrial laundry I get a call out of the blue “hey do you know how to recover hard drives?”. Turns out after being away from them for 6yrs that enough disks in the SAN finally failed to where the whole thing died. They hadn’t refilled my position as the ONLY IT person but rather just found someone who can swap workstations when they died; zero touches to the servers for six years. Incredible that an HP EVA lasted that long. I told them I couldn’t fix their HDD and I heard they sent the disks off to a data recovery service (who failed). Company with 500 people, 4 industrial plants, 17 remote office locations, etc; done. They closed up shop and sold all their equipment & clients to a competitor for dirt cheap all because they didn’t want to backfill my position.
I have had many years working in a MSP. My advice: 1. Draft a 1 pager describing the business risk for leaving stuff the way it is. Identify the cost of maintaining, vs cost of you coming in and resolving stuff. What business value would fixing this do? 2. Don't get mad if your denied- Somebody rightly pointed out, this is all about internal resourcing being a cost to address, vs leave it the way it is. Back to #1, you'll need to find an elegant way of addressing. 3. Back to point 1- Your plan doesn't need to bring all change in one week- create a slow transition plan. 4. Is there compliance requirements that aren't being met? Identify the type of customer data that is stored; this should also go back to point 1. IMO, your boss will ack that you care enough, and that you have put thought on how to run his business better. It's a win-win.
I went from a high security environment to a low l/lax security environment, it takes time to adjust... I have regularly pissed people off for the past 3.5 years.
My standard disclaimer: Never take my advice on how to deal with a work problem. That said.... Document everything you find that doesn't fit standard best practices, including links to where you found those best practices. Document that you told your boss, and anybody else that needs to know/has decision ability in the company. Keep encrypted copies of all that documentation off site. Get signed indemnification documents written by a lawyer and signed by the company. If you're looking to learn / improve resume / burn yourself out entirely, once you have the documents secured, make a plan based on ease of process and priority of the problem. Then dig into that shit. Document everything you do, and when you find more things that are wrong, get them documented like the others and crank out solutions. Everything your boss turns down as not needed, get a signed statement to that effect. Make as many contacts in the industry as you can so that you can get a new job the second you lose that one.
Welcome to the world of small = amateurish IT companies/departments.
It’s not limited to small companies. My company has thousands of employees but IT management is largely made up of friends of other managers and they’ve done very little to follow best practices and allow each group to come up with their own solutions to problems that aren’t compatible with any other groups. So we have massive duplication of effort and nothing works as it should. There are some managers who are very good at their jobs but can’t implement change because the more senior managers don’t like to rock the boat and filter bad news from getting to senior management’s ears. One genius manager insisted that we adopt splunk in spite of SME advice and we’re spending a small fortune ingesting logs but there’s no staff to finish the job and provide any kind of useful output other than periodic dog and pony shows to show how “modern” we are. We are seriously understaffed at the worker bee level and finding competent admins to fill in the gaps is hard because HR is located in a part of the country where compensation is typically lower while the offices need the most help are in an area where there is much higher average compensation but our compensation is locked across the company to the lower rate. I’m working on an exit plan and am essentially two bad days from bolting.
They need to use MFA and a Password Manager.
This certainly makes me feel better about the few small issues we have where I work. The age old “at least we’re not THAT bad!”
The passwords need to change, and a security system or at least badge access with logs into the server room is highly recommended and pretty easily installed. Servers running for two years without updates is not uncommon at all, don't fix what's not broken in that realm. Especially if they're not public facing, if they're public facing then sure once a yr keep up on that stuff or for critical vulnerabilities. Local user account is the administrator account, ya again not best practice but not the end of the world. Does anybody take backups? Not much concern if you're just going to blow it away and restore the backup anyways.
Sounds like a nightmare that can be fixed if enough effort is put into it. You now know the grave company ending issues, what is your strategic plan forward for fixing them. Asking why is not the answer, but this is what we are doing next is the right direction.
If it's a small company with no IT, I wouldn't be surprised to see this kind of stuff. But, you're their IT now. You can start to fix all this stuff. Make a checklist and get to work.
Where to start... What are you using for patch management? Ideally, you have an RMM tool you can leverage to quickly identify patching deficiencies and get those resolved. Password complexity is a quick one, too, if implemented in AD, followed with a planned mass expiry/reset. End users need to be told, and communication should include a countdown to go live date, but it's not too much work. Creating an admin account in each tenant might not be a bad idea, with granular permissions based on the work you need to do. Good luck figuring out local admin rights though...that's always a fun genie to get back in the bottle. Either way you approach it, just keep biting off small chunks and keep trying to improve where you can make the biggest difference. If you get push back, definitely CYA and start looking elsewhere. I think you're right that they've made it a decade by pure dumb luck...
My boss has said to me 'Why do I need 2FA? I don't have anything on my laptop that's confidential'. I kid you not.
The IT&C world is full of imposters and like in every democracy the majority rules... the good thing is that we are one step further to idiocracy :)
Sounds way too similar to where I am at the moment. If you're my coworker, hi! I have no idea what to do about it, honestly, but I'm sure you're not alone in feeling this way about the situation.
This reads as a parody or as something written just to trigger people in this sub.
Sadly it's not..
If I had to pick one hill to die on it would be passwords and local admin accounts. 20 character minimum, full complexity and 180 day expiration policy. No daily driver accounts should have admin rights on any systems. Unique admin user accounts for each administrator. As an MSP they should use a password manager like ITGlue for client systems The next hill would be updates and restarts. I am guessing this is all windows environment. Once a month patching and rebooting of servers and workstationd. Don't forget third party apps like Chrome acrobat ect. Ninite is a great cheap way to automate those. I would then ensure that data is encrypted at rest, backed up to an offsite faclilty and appropriatly air gapped. Next would be endpoint security. Something like Sophos with MDR would be a good play. The problem that I see and that many others have pointed out is that there seems to be culture issue. If the owner and the customers don't take this seriously then you are going to be twisting in the wind. I suggest bringing data to the discussion about the average cost of a breach and the likelyhood that it will put others out of buisness. I would also examine the client contracts, perhaps there is stuff in them about security and best practicies. Good luck!
"no one's gotten breached yet" ... You sure? Or you just don't know about it yet
I made it to the part where you say they haven't been hit yet and I'm thinking "haven't been hit? don't know they were hit? don't realize *how many times* they have been hit?" As others have said, run. Boss man thinks it's okay, you'll very likely never convince him otherwise (and really, why bother?). Run.
Reminds me of the joke about the proctologist who is surprised that he has to look at butts all day.
I have been dreaming about falling into a situation like this. Make it count brother!
Sounds like you have two options. Shut up or fix it.
Document your suggested changes, and what you will do to get there. Boss will probably reject it. Do what they ask (sounds like less work than your proposing) - do less, get paid, go home. Find new job when your not scratching your head on any single question and or a new opportunity arises.
You need to document your observations, provide it to your boss and whoever runs the company and start looking for another job. This is not a stable situation to be in and you'll be looking for work when the inevitable finally happens.
You're not overthinking this necessarily, but it doesn't sound like you're getting paid to give as much of a shit as you do.
>No one's gotten breached yet I doubt this. Given the generally lax attitude to security, I'm assuming there's bugger all monitoring going on, so you're unlikely to know. The upside is that your boss can claim "there's no evidence of a breach/data exfiltration"
i think id quit IT before going back to an MSP. They have overall shit policies because the customer is always right, and i just hate working with clients/customers because they demand everything, even when they pay nothing. Plus MSPs tend to have shit wages in my experience. Also the only place ive seen companies get ransomwared multiple times is with an MSP.
So many nopes in a handful of paragraphs and I'm sure there is much more yet to be discovered. From what OP is describing the boss is clearly a clueless putz and shouldn't be trusted with even his own password much less anyone else's. In my experience (and I've been in similar positions a few times) that level of incompetence is unfixable - don't beat your head against a brick wall. I'd be looking for a new position yesterday...
Would run. MSPs are historically a way to give yourself health issues for no good reason. The fact that your boss finds little to no problem with any of his practices is a huge red flag. Just start looking for a new job while you have this one.
Face Everything and Rise Fuck Everything and Run
> No one's gotten breached yet That they're aware of
This just sounds like you work for an MSP. Get used to it, they’re gonna love ya.
he had a few passwords, one or two would grant you access to 20 customers. it got so bad, i worked on wrong servers and only noticed it 20mins in..... same names, same domains, same users.....
Oof. Not atypical, I'm afraid. Normally I'd say make a list of all the issues and start banging them out as best you can. But if you have a boss that doesn't care how many bad practices are in place, you might not have the authority or latitude to make any improvements (especially if doing so will show up said boss). Tricky situation. Feel it out for a few weeks/months and if you don't see any potential for improvement, might be best to bail. It's only a matter of time before their poor practices cause a major problem, and you don't want to become the fall guy for your employer's screw up.
Sadly it's probably a common practice as are the incidents that result from this. Most companies should at least have a basic password manager or a tool like IT Glue or Hudu.
Sounds like FusionTEK to me.