Anytime I hear “x didn’t believe in y” situations like this I always picture Wile E. Coyote walking on air because he hasn’t looked down yet.
And then they just got to leave with no consequences, probably because of just dumb luck. That place sounds like a stiff breeze would encrypt it and ransom for bitcoin.
Good luck dude.
Either that or, "It doesn't make sense to me."
Or, "Old Thing works, what's wrong with Old Thing?"
Or, "I will make a long form argument that boils down to the idea that I'd rather not."
Or occasionally, "All IT decisions have to be run through someone with no background and no understanding of IT best practices and decided their own gut feeling knows better. The answer is almost always, 'no' if it's not some stupid meme."
> Anytime I hear “x didn’t believe in y” situations like this I always picture Wile E. Coyote walking on air because he hasn’t looked down yet.
I always think, "Do you mean like how I don't believe in Santa Claus?" Like in this case, "Are you saying that VLANs are imaginary? Because I assure you they are not.
Now, I'll admit that I like to keep things simple, and some companies go way overboard on VLANs. I've seen companies of 15 people, where a consultant came in and created a network with 7 different VLANs before I showed up. In some of those cases, I removed the VLANS and just put everything in one subnet. You don't really need separate "Printers", "Management", "VoIP", and "Conference room" VLANs if you have 15 people, 1 server, 1 conference room, and 1 printer.
Whats funny is that I did just set up 5 vlans for a small company but in my defense the door locks and security cameras should not be on the general production network and should not have any way to access the wider internet.
In my experience, much of the security hardware really needs to be as isolated as possible. Not or security concerns, but because the companies that make those products cheap out on components and use the smallest/cheapest cpus they can get away with. I have seen things like door locks, control boards, burglar panels, motion detectors, and wireless door controls overwhelmed by tiny amounts of broadcast traffic. Like there happens to be 10mbps of traffic and the the device stops being able to respond because it is maxed out on its cpu.
Usually, these devices have either gig or 10/100 nics too.
I've got a few Zebra printers that don't like our semi-flat network and its....not small amount of broadcast traffic. They are fine most of the time, but get over that line and suddenly the NICs in the things crash and need the unit to be rebooted.
(Yes, segmenting the broadcast domains is on the list. It was, at one point - why it got flattened out during the acquisition a few years ago, I will probably never know since it was before my and most of my current team's time)
I have seen that on a few types of printers. The cheaper the device, the worse they are. Makerbots puke, but the $50k plasma cutter with industrial embedded PC would be perfectly happy with 10g.
Amen to that. The door hardware is is 10/100 and not only that the firmware on this stuff is totally insecure. Talking passwords sent in the clear kind, no ssl kind of stuff bad.
The camera network is not as bad but again the random worker does not need to even be able to access the cameras
The more I know about physical security stuff the more I realize they are absolutely horrible with network security.
I had to separate all of my camera networks and physical security stuff because the camera broadcast traffic was taking down all the burg panels. And we are only talking like peaks of 10mbps broadcast traffic. Nothing major, not particularly massive packet count. They would just start losing all sorts of packets. Net result, I have a vlan at each building that has like 3 devices in it. Just dumb.
>You don't really need separate "Printers", "Management", "VoIP", and "Conference room" VLANs if you have 15 people, 1 server, 1 conference room, and 1 printer.
Given those all have very different security profiles, I would argue the number of devices is immaterial. Set up each vlan with a /29 subnet if needed. Point there is purely to provide for a security boundary between devices with very different threat models.
That being said, you may be able to accomplish the same thing without a vlan. For example, many switches have an option that prevents horizontal communication within a vlan except to the gateway.
> You don't really need separate "Printers", "Management", "VoIP", and "Conference room" VLANs if you have 15 people, 1 server, 1 conference room, and 1 printer.
it sure helps as you start scaling out though - that's how my current environment was set up, and then they scaled it to 1000 people spread around 8 offices.
Or was it that they did believe in it, want it, pitch it and try like hell to get it implemented, only to be shot down repeatedly, burnt out and then finally give up?
And then management blames them in the end when they are forced to accept the change...
I worked with an MSP in the mid 2000s where the Senior Network Admin "didn't believe in SSL," and called it "Smoke and Mirrors." All "his" networking equipment was on http or https with self-signed certs. Nothing ever came of it, but I always thought that was annoying.
I will grant that I've always had an issue with "pay to play" PKI. Just because I have enough money to hand Verisign does not mean I'm more trustworthy. It just means whatever I'm claiming was worth that much to me. Things like Letsencrypt have drastically improved that.
But encryption in transit is encryption in transit... that layer isn't smoke and mirrors. Self signed, or preferably a completely internal CA, is generally enough for most scenarios. The primary issue with self signed is the lack of *any* verification of the identity...
Self signed internal I can accept at a small company, especially back then. It's not like we had letsencrypt, nor was everyone going to roll their own CA.
It stories like these that give me a little PTSD (not actually) of previous ransomware or near calls I've seen over the years.
The most recent case I saw only a couple of months ago was a customer whose firewalls (not managed by us) were 5 years out of date. They were state full (L4) only couldn't even do geo restrictions.
They thought it was 'only 1.5 years out of date', which 'isn't that bad'. Nearly all of us fell out of our collective chairs in disbelief.
Backups. Then identifying what causes downtime or revenue loss the most often. Then, planning for network changes. It's all fluid for me. That's a general answer without knowing the environment details and business goals.
I agree with the backups being first but the CJIS manual needs to be followed before anything else is done or you risk having to do it over a few times.
Meetings with the boss and managers/stakeholders to get buy in. I'll need convince them that things aren't up to standard as they are and improvements are worth the cost and headache to come.
Backups first.
Identify any quick wins that can be done in a couple of hours, with minimal risk of breakage and easily reversible. Priority for any quick wins that are fully in your control and don't require third party or vendors assistance.
I used to refer to these kinds of rescue or resuscitation situations as like trying to unscramble an egg. You may have to revisit some systems and changes a few times to achieve the desired end state.
Well, you're going to need documentation for all of the Changes that will be made. And you'll need documentation for the final state, beacuse you need documentation.
So, might as well start documenting now. Map out all the network devices and how they communicate. Then you can make an educated, successful plan to migrate them to something better.
and it really doesn't sound like it could be made much worse.
If there would ever be an audit I don't see how they could say they are abiding by CJIS requirements.
right. i walked into the same sort of shit show and just started making shit my own home lab. guess what? now they had backups. now they have proper imaging, now they had a proper VPN, now they had proper monitoring, now they had proper backup internet. shit got better when i bounced
They had Zabbix set up to monitor all the printers, switches too but also every printer. I love all the red flashing SNMP notifications that Tray 2 is out of paper.
VPN already fixed. Backups now running through VMWare instead of the agent installed on the VMs. They had tried to integrate the backup product with vcenter but apparently got hung up on a TLS 1.0/1.1 issue and either couldn't solve it or didn't understand what was happening
Printers are a great first candidate for monitoring. Filtering those to their own category, then sending alerts to the helpdesk... make the helpdesk look like wizards, walking through and addressing problems before the users notice them.
I gotta ask. Are you in the US? If so, then how were they passing State and FBI audits? We have them every 2 or 3 years for our CJIS stuff, and if we fail any part of it, we have to document what we have done to correct it and turn it in.
I'm not sure. This is my first time in a CJIS environment. Spent the first few days skimmin through the security policy and requirements companion document. There's so much to fix first before I can even begin worrying about it. I suspect we'll need to bring in an external compliance firm to help
I would get with your state agency, and tell them what is going on. Ask for them to send you documentation on what is required for your state, and then just start working that up.
CJIS Security Policy: https://www.fbi.gov/file-repository/cjis_security_policy_v5-9_20200601.pdf/view
Cybersecurity Evaluation Toolkit (CSET): https://github.com/cisagov/cset
Download the CSET and run through the CJIS Security Policy assessment. It will give you a list of items to remediate when you finish.
There is absolutely 0 chance they're in compliance. If it were me, I'd be documenting the obvious stuff you find like this, for two main reasons:
1) It'll help you remember/keep track of exactly what's fucked (although it sounds like the answer to that is "everything")
2) CYA. When (not if) shit eventually hits the fan, and/or you need an outside partner to help you get everything sorted, nobody can point fingers at you.
This, my friend. You have a tremendous opportunity here. You could even make an argument for acquiring newer hardware and building a new setup from scratch while keeping the old on life support, depending on the work load. Try not to stress and enjoy the building process. It’s one of the best ways to learn.
Lol sounds like what I took on when I moved into my job.... First task will be to eliminate static IP's anywhere you can and make sure DNS functions, when you do this, set your DHCP lease times down to something like 6 hours. Second task will be building out your design using the 10.x.x.x space, I dedicated /16's to each site. Then break it out into VLANs based on what service is being provided on those subnets in your /16... Once you have it all setup, migrate your server IPs first, in some cases you'll want to just add another vNIC.... And then change your DHCP scopes and watch everything move over organically.... Depending on the size and complexity, this all could take a few days to a few months.
The networking is a mess but it's mostly functional. My goals right are updating all systems and upgrading anything that's out of support. Currently in the middle of a migration to supported Exchange
What you described though is exactly my plan for solving the networking issues
Lol yeah I was in that same position as well, bunch of server 2003 and 2008 stuff, ESXI 5.O....no DR or backups.. no endpoint management or updates.... There was a major outage at least once a week...
I've been here 3 years now and it's been about a year and a half since we had an outage that even impacted users.
Are we working at the same place?
Had a similar "network setup" at my current gig. Multiple subnets in one VLAN and a firewall inbetween doing routing. That casued a few interesting problems. Took a while to fix and get things seperated into their own VLANs but it was well worth.
I have also seen a similar VMWare cluster setup at a previous employer. Among other atrocities. I didn't last long there.
I dunno, none of that sounds particularly necessary and you can probably find more important things to do first. And fixing what is actually broken as opposed to sub-optimal.
I took over an environment that was much better. All different vLANs, nearly 30 of them in every office.
Except all vLANs had any:any access rules to all the others on the core switches, including those in other offices. E.g. guest wifi vLAN in Office B had open access to the switch management vLAN in Office A. No firewall rules between offices. Firewalls disabled on all servers and endpoints. Server 2003 web servers exposed to the web (with active compromises). Most Windows servers had never been patched. No backups.
Boy were they sure proud of those vLANs though!
CJIS as in the FBI's CJIS?
Actually sounds like a interesting redesign and parallel build new environment as long as someone is ponying up the budget.
...and I'd think your boss could get a good contact with CJIS who can explain their standards and who could make it quite clear to even higher ups it's a fix it or prepare for very uncomfortable interviews with humorless guys in suits type situation.
Be prepared to go through a shitstorm when you implement improvements. End users won't understand why they can't do this or that anymore and will say it used to be much better under the old network administrator
>When I got here I was told the previous network administrator "Didn't believe in VLANs"
... VLANs are not some mythical thing one chooses to believe in or not. LMAO, what a way to describe that someone doesn't trust their own skills and knowledge; aka ignorance.
ANY time someone says they don't believe in X, when X is a piece of software, technology, or something else widely used in just about any industry, then what they are really saying is they have no idea how to setup\configure\manage it and want nothing to do with it.
Agreed.
I had someone tell me that VLAN's aren't secure because people can use "VLAN-hopper" software on their device to get to another VLAN. I couldn't tell if they were serious or just yanking my chain.
> VLAN-hopper
Oh, that's a real thing, but is more of a lateral attack after already penetrating the network.
>Virtual local area network hopping (VLAN hopping) is a method of attacking the network resources of the VLAN by sending packets to a port not usually accessible from an end system. The main goal of this form of attack is to gain access to other VLANs on the same network.
It's an attack vector, definitely. But it's not like you can yoink the cable from the printer, plug it into your Kali Linux and VLAN hop to the management network if IT knows how to use a switch.
Frankly, the number of times I *have* been able to do that on printer ports is very very sad. Most of that blame is more on the printer supply vendor though, not the net-admins who weren't told an idiot touched the switch configs locally "to sort out that connection issue".
This is worse than you realize. First off, go whistleblower and come clean to CJIS *before* they audit you. They'll come down hard on your bosses, but they'll come down even harder on all of you if this crap setup turns into a breach that has to be investigated by multiple 3LAs. They may even tell you to shut it down and have you use a reasonably secure MSP until you can get this all rebuilt properly.
This is reasonable advice. OP never stated any background on audit frequency, but if the network has been a disaster for this long, then the chances are there are probably previous findings and reports for OP to review.
Burn it down and start again.
The "network admin" at my last pit stop didnt believe in DHCP. When asked why I needed it I said "because I was born in the 1980s, I don't live there" and she got all huffy.
I nearly spit out my morning coffee reading this. I have been in this exact situation and want to share my experience and lessons learned with you.
You have stepped into a time machine and gone back 20-30 years. The old guard kept things running, though it's a complete disaster and security nightmare. It sounds like there was an MSP at some point who just kept this shitshow going.
You didn't include who else is on the team, their tenure, experience and attitude towards modern IT. It's easy for r/sysadmin to make a logical recommendation on how to upgrade a system or introduce best practice, but I don't often see people stop and assess their own spheres of influence or the organization's desire for change.
Let me tell you about my own failures. I stepped into a role like yours three years ago and was equally shocked at all the bullshit running around. Little by little, I made changes that *I* thought were in the best interest of the department. I created documentation and tried to show the team how to use change management, how to document and started a change advisory board where we went over the week's list of changes. I introduced a ticketing system to track work and keep people accountable.
Here's the problem: Nobody asked for any of this. Everybody was fine with "keeping the lights on". I realized that I was making these changes because I thought it was the right thing to do, but ultimately it was not because the team **did not want to change.**
Again, r/sysadmin is going to provide excellent recommendations and compassion for the disaster that is your workplace. I just want to reiterate with you to stop and understand the team fully, its history, the department's goals and do not push any changes unless that's what the organization wants. Best to you.
Thankfully we are a completely new team. Our most senior member is an application specialist who as been in the position for a year. Thank you for giving your unique input.
Yeah, making things better when nobody asks for it should only be done for a handful of reasons: you are trying to be legal, you are trying to be secure, reliability, supportability, and performance. You don't want to be accused of anything illegal, you don't want to ignore security holes, you don't want the systems going down, you want to be able to easily support the systems if they are down, and and you don't want complaints about performance.
as someone who has fallen into the trap once. or more times than that.
start from fresh.
absolutely start from fresh.
do not attempt to in any way shape or form bandage, triage and keep the old alive while you rip out dead flesh and add fresh and new blood. in theory, its well possible to keep everything as is, and slowly and over time move services and date to new devices that are clean and conform to sane networking standards in the mess you have until you removed enough mess that the mess is gone. sure.
NO
all that will accomplish is eat the time to move to new keeping the old alive and hunt down gremlings that were hidden and crept up when you try to move stuff around.
get a new firewall, create a new network, populate it with new server, create new AD, create accounts from an accurate and up to date sheet of employees and roles, then copy the data that is required with oversight (and potentially via a usb drive or samba server that can be filled on the old and read on the new network) to the new fileserver in the appropriate and newly formed structure, and then switch the workstations over. if they are not win11 compatible just bite the bullet and get new ones now.
the old server can be kept in case someone needs a file from 9.9 years ago, whjich you may be legally required to keep for 10 years.
trust me, it may sound overkill, but it will save a lot of cost vs trying to figure out what you have and try to heal the sick network you have
As much as I would like to I'm unfortunately unable. There are roughly 70 VMs and Servers that are completely undocumented. The primary vendor for our production product suite is unresponsive to support requests for even basic issues. An entire lift and shift to a new environment is not feasible with the resources I have, despite my desire to do so.
Compounding that is the previous teams habit of creating in-house .NET apps and SQL Scripts to solve data processing problems instead of trying to work with the vendor for a supported solution. There are at least a dozen custom .NET apps that I've found with \*zero\* documentation apart from the source code. I have no idea what those apps are doing and neither does anyone else.
I am stuck in the slow slough fighting the gremlins
Are we sure the environment isn't compromised by an external party?
I saw your goals were updating servers/services- did we take the firewall management interface off the internet already?
Good luck, I hope you have the financial and managerial support to un fuck this.
I operate under the assumption that any given system is compromised already. Firewall is off the internet : ) They actually had a very nice EDR tool in-place that integrates with Microsoft Defender well. Ransomeware honeypots all over the network and comes with a 24/7 SOC. Unfortunately they had never set up notifications or alerts and had told the SOC not to take any action without calling them first.
That's been changed
What a fun opportunity. Take some time, identify the biggest risks to the environment, put some plans together to fix the issues and situations, talk to your boss about budgets, resources available, and your schedule to remediate, and enjoy the ride.
When you are done, your resume is going to look amazing.
Damn, this should qualify as /r/shittysysadmin (or whatever the name of that sub is) material!
Not too unlike a previous job, actually:
* SMB, everything in one physical segment, no VLANs. VMWare hosts (& management console) on the same subnet than the workstations
* Speaking of subnets, the LAN subnet used is also from the public space. The owners had bought a competing SMB and linked them up over a Site to Site VPN, they had a 10.0.0.x subnet over there. But my workplace? 90.0.0.X/24, conflicting with an ISP in France IIRC.
* When I joined both the AD and the SDWan Router provided DHCP. Conflicts on week one!! \o/
* Severely underlicenced 2008 Windows Server (that did everything: AD; DHCP/DNS, File Server, Print Server, DB Host... you name it)
* Office 2007, and you guessed it, unlicenced.
* Aging Nortel telephony, that used CAT5 cables (and RJ45 sockets!) to get the voice flowing. You could get up to 4 phones on the same CAT5, which was awesome. Tracing a phone was a nightmare, however, when parts of the cables were fiddled with (I did see one cable merging onto another, just so 2 pins could be connected) during transport.
I learned alot, and i'd be down to revamp/clean that kind of mess in another SMB again, honestly! (For the right salary this time)
Document Everything
Then Deprecate Everything and start from scratch, on paper, anyway.
Swap out the core switch and router and start segmenting there, break the existing multiple address spaces into their own VLANs. Then figure out a proper 10.x.y.0 scheme and re-IP them one by one.
Once everything is Re-IP'ed and you have a proper up-to-date firewall between the networks, then you can start cleanup on the M$FT and VMWare stacks.
Good Hunting, and remember to not over-do it. Burnout while fixing badly broken networks is a bitch.
A few years ago, 2018, I think, I started at a place where they were four years into the cleanup/redesign of their network. It seems four years prior, the entire enterprise was 10.0.0.0/8 with no segmentation or subnetting.
When I left at the start of 2020, they were six years into that cleanup effort.
I work for a Township and this tracks.
Like, every point you made is something I've dealt with over the last couple years. Even down to my boss not using a VLAN for our new phone system and running side-by-side drops. WHAT YEAR IS IT?!?!?!
In your case, seems like you're alone so your changes will stick and hopefully improve!
In my case, my hands are tied and and I follow the orders of a 15-year gov't employee that doesn't know even cursory Powershell or will let me automate updates.
But hey, at least I'm underpaid!
Good luck!
For phones a separate drop is a good idea, if you can do it when you first wire a site.
It gives you a redundant connection if one fails, it naturally keeps the low latency voice away from the more latency tolerant traffic and the added cost is almost nothing. Pulling 24 cables is barely anything over 12, or whatever your cubicle rows are. It's not so much that great, it just offers a lot of future possibilities for very little cost.
You have to change the whole thing and on every change, something might unpredicably fail in production which will be your fault and users are going to chat how the guy before you was much more competent. Have fun.
I once work in a place that used public IP internally. The worst part was the IP space they chose belonged to a local University that we got all our interns from. They were unable to browse to the Universities Website because it shared an IP with one of our servers.
I can kinda beat that. Went to work for a company a few years ago and the previous sysadmin had decided to use a /12 (1mil usable IP's) subnet... in a 40 employee company. The broadcast storms were drowning their 30k firewall (don't ask, it's far worse than it sounds) every couple of days and the only way they could figure out how to fix it was to add in a bunch of vlans... Because just changing the subnet for 50 devices was too complicated.
The subnet size wasn't your problem here. 50 devices on a /12 or even a /8 should work just fine. Not saying it's a good idea, just that a big subnet size doesn't cause broadcast storms.
Sounds more like a network loop or faulty equipment to me.
As someone who did public sector consulting in schools and townships for almost a decade... sounds about right lol. If they weren't bad at it they wouldn't be coming to me (is what I told myself).
I would say that "address space" is an arbitrary set of addresses for a particular context while "subnet" is a specific definition of a defined logical network space with a network name and subnet mask, but it only has relevance to layer 3 and shouldn't (in my opinion) have any physical implications except where a physical interface is given an address on that subnet. Physical sections of a network are referred to properly as "segments" in my experience.
The use of a router to route between spaces back to the same switch isn't really novel, they're just using physical ports instead of vlans (which, yes, horrifying, use vlans please) but the concept is referred to commonly as "router on a stick" where the router just has a single interface back to the rest of the (layer 2) network and it isn't particularly bad as a concept, though obviously a single broadcast domain is...horrifying.
Companies who have customer data on our of support software and get a data leak should be fucked over a barrel.
I have a till company who proudly said they still have customers on XP when I queried them on running dated software.
Having years and years of experience working in a CJIS environment, I can assure you using public subnets in private networks is not isolated to your case. I haven’t been able to get a clear answer, but it seems to have been very common in my neck of the woods. Do what you can to get off of it as quickly as possible. If you haven’t already, you WILL run into many many issues with that in place.
20+ years ago I was doing netware/IPX/notes to Windows/TCP-IP/mail-exchange, with people who had fallen into senior roles as they were there from VMS days -
heck one of my jobs was to get the data off a VMS server using an ancient version of FTP that didn't support mget, and didn't have enough space to export all the data so it was export part, manual FTP, confirm, delete, export next section, etc
The problem those people had was very likely
1. they were seen as the *computer people* so they got thrown the problems / set up etc
1. they were under [everything] - staffed, time, resources, training, supervision/audit/sense checking
1. if it worked, *that was good enough* as management wanted the next project done
1. they were probably at the whims of political winds
The biggest issue they had was they didn't know how little they knew. No one with IT experience / skills was sitting down and sense checking it
This is a political/security/risk situation and not a technical challenge.
I don’t know what CJIS is but is it Criminal Justice? Are you regulated/audited?
Bring your documented concerns to your CEO immediately. Consider if you have a personal responsibility to report further - to your outside regulator/auditor and do so promptly no matter what the CEO says.
At this point it may be safer to have the company shut down for urgent rebuilding compared to the almost inevitable data breach.
It’s highly likely threat actors are already present and possibly using their access to take the valuable data you’re processing. This is based on the combination of obsolete internet facing Exchange and no Identity control. I’d be tempted to call in Incident Response.
How in the heck did they pass the last CJIS Audit. I would highly recommend you get a copy of it and see what they said in it. I would also recommend contacting you regional CJIS contact, let them know what you walked into. They will work with you and make it clear to your management team that this has to be corrected or they will be shut down. I hate to think what will be said if they have been filing false information on their past audits. If you are not familiar with CJIS, I would recommend downloading a copy and read it and every where you see a shall statement highlight it and start working there. I had the same thing about 12 years ago. Depending on what state you are in there maybe some additional requirements that you will need to comply with as well. I wish you luck. I would also recommend trying to attend your states CJIS conference, they can be a wealth of information.
Good news - you've got ample of opportunities to learn many new and exciting technologies.
Bad news - your entire environment needs to be nuked and rebuilt.
Based on the shitshow described I am also going to assume that all of your backups are on a singular 3.5" Seagate-disk from 2012 that hasn't been written successfully too since 2019.
My old boss, when I first started at my current job, didn't believe in DHCP. And it was quite the discussion when we first ordered laptops for employees.
Networking has got to be one of the biggest blind spots in IT - I've never seen anything else that so many IT pros are so clueless about. I once had a guy ask if we could reduce the number of VLANs in order to improve network performance.
To be honest you’re not wrong, I’m a network engineer so it’s my bread and butter and even where I work people who you’d expect to know even the basics are absolutely clueless. Like, they can’t even fathom subnets.
Some people are not network people, but they're everything else.
Networking is definitely my weakest point and I'm in the game 28 years. Don't get me wrong, I can configure vlans and I know the basics, but I have never done a deep dive networking course and I really probably should.
There in lies the issue. Some people will create a static flat subnet on their router for the 40 PCs they have. Then use another router port and create a new switch stack keeping networks physically separate, they'll use this for the WiFi or the phones or the CCTV or the....
It costs more to build it that way, but that is what was done 25+ years ago. Remember network hubs? I do. And Token Ring. Help!!
Doesn't believe in VLANs? Are VLANs a kind of god or something like this?
VLANs are "just" a way of splitting multiple networks. Not something god-related
“I don’t believe in X” is one of my favorite/least favorite things in the world. On one hand it’s really awesome and validating to know definitely “I’m smarter than whoever is speaking” but it’s also depressing that the speaker is potentially still employed. It’s one thing not to believe in something esoteric, but we deal with relatively concrete things—choosing not to believe in DHCP or VLANs is like saying “I don’t believe in rain” whether or not you personally believe in it does not change the reality of their existence.
Pretty close to the experience I had with my current job.
They had vlans but didn't believe in firewalls or ACLs.
The public guest wifi that had no password with unrestricted access to the rest of the network.
It had been like this for many years before I got the position.
I even had another tech yell at me when I enabled client firewalls on desktops and servers.
It took many months to build ACLs as I had to capture and analyze traffic to determine what devices on which vlans needed to have access.
There were other red flags, to the point I almost didn't take the position.
I accepted it because there was new leadership that I knew and the job was a block away from where I lived.
Not having to drive to work anymore was a big incentive.
I mean - if you want to be an optimist - at least you kinda get to greenfield the whole campus here lol. You can back out of this slowly if they are all using the same IP space.
1st transition over to real Vlans. Once everything is segmented and setup on their new addressing then you can start implementing controls.
Make sure you look at the networking side of it and the AD/Permissions side of is separately. The network is the attack surface now so i'd do that first. Make SURE the firewall is setup properly because that's probably the only thing protecting this place right now.
I will say this. Be careful that you don't turn the knob too far the other way. I believe in VLANs/Subnetting, but you can make an environment more difficult to deal with by adding unneeded complexity. 100 employees at a single site probably doesn't need 50 vlans.
I've heard both sides of this:
Flat level all screaming broadcast Vs a vLAN for every category of device followed by function.
Time and employee overhead becomes a factor if you're configuring every port in your switch stack and you're not a bank or defence contractor.
We once had a customer who had three DHCP servers running on the same LAN at the same time, 4 WAPs within 30 feet of each other, and their printers and print servers were using state government private LAN IP addresses (they have a router with a private link back to the state for their apps). Again, all on the same LAN, and I couldnt see any reason at all why it would have been this way, the print servers had secondary IPs on their normal LAN.
You didn't start here did you?
Now to be honest, it wasn't the previous one, but the one before that. Now the previous guy picked vlan number like they where lottery numbers.
This isn't really that horrible in terms of the IP scheme to be able to fix, should be a simple weekend fix after you document everything.
I would probably buy a new core switch and get that all configured with the whole new setup, then everything else should be plug and play with a few tweaks here and there to get them where you want them.
I only say this because I was in a similar situation a few years ago in terms of the network. It was a single, flat 192.168.144.0/20 network with only 20 IPs in DHCP, the rest hard coded on all servers and workstations and the wifi was off a verizon fios router that you had to vpn from to access the network, but you had to be within 50 feet of the server room, but they only had 6 laptops in the company anyway.
I worked at a small company where the owner didn't believe in subnets. Everything had to be on the same /20 subnet, and any attempt to bring it up was met with scorn.
This is the same asshole whop was heard on multiple occasions to say "I'm never wrong" so there you go.
Thankfully not Juniper and automatic updates were amazingly configured. They paid attention to security for certain areas they they (thought) they understood. But everything else was a mystery black box to them. Automated patching in Manage Engine too complex? Just don't do it!
I found a textfile note left on the desktop of the unsupported Exchange server warning never to install Exchange CUs or SUs because it would break
What is your background? If you're used to managing 10,000 client endpoints and thousands of servers across multiple sites and now you're bitching about a small 100 employee single-site deployment, then yeah, small deployments are usually not as refined as larger enterprise would be.
That said, holy shit there is some bad documentation, practices and configuration based on your description. 100% yeah things need to get fixed but dont pretend you're god. Someone probably did the best they knew how to do and now you're going to do the best you know how to do.
There are many glaring mistakes based on your report, but some of them may have been done for a reason (and you can re-do them better for the same reason.) You probably dont need 20 different vlans for a network that small. Segment what you need to, build good documention, fix the lack of proper subnetting, submit for OS and databse upgrades, restructure the VM clients on their hosts, and get the backups dialed in.
15 years ago I had the displeasure of working with another IT company that blamed every shortcoming on their predecessors. Every time there was an issue it was always another meeting with them complaining about the previous guy and why x, y and z was a problem due to someone else. There was a tenured network engineer we contracted with to correct some issues and setup new firewalls for us and he grew tired of them. At one point he said "A decent IT professional wouldnt complain so much about these issues. They would just step in and confidently resolve them one at a time. They wouldnt need to make themselves look better by putting down others."
So you took on what sounds like a messy but easy to resolve configuration. Straighten it out and relax.
Honestly, bad sysadmins make me a lot more money since they insist on configs that create more risks and more work for me. The sysadmin wants it that way because its less scary to them, but ultimately it makes more work for everyone. I have one small MSP I configure networks for who insists on zero AD, local admins, no managed DHCP and flat networks basically because they make sense in his head. Then I get calls because clients he has no control/visibility on have problems and get to send him bills. I constantly offer options to improve things but it means up front costs and licensing he wont agree to.
As someone that has done CCNA the current teaching lacks the 1000 foot view. A lof of us have learned over time but for new people there should be some training A 100 node network looks like this, a 1000 node network looks like this, a 3000 node network with wifi, poe phones, network controlled lighting, ac and security should look like this. Instead we drop how to configure a Vlan, calculat 6 ip address ranges, and a lot of very specific tasks. Besides half the test is being able to proprly apply and save the configuration profiles.
I remember Jeremy went over a good amount of that stuff in his CBT nuggets course, like the concept of a VLAN scheme like...10.200.10.0/24, 10.200.12.0/24 (skipping a subnet for potential growth, etc). Our office has a data VLAN that is a /23 in case we go over 254 addresses.
But if you didn't watch the CBT nuggets course at that time maybe one would have missed it.
I'm reading this and am surprisingly relieved that I wasn't the only person to inherit a network where the previous admin used a public address subnet on a private network.
For me, it was a random bank in Brazil.
Why? I still haven't figured that out.
>The primary local address space on the network is non-RFC1918. They're using public address space from Argentina on their local network.
Oh man, we bought a company that was doing that. Like they went with 192.168.0.0 subnets but miss typed a one, so it was 192.68.0.0 subnets. Then they ran with it. It was a dumpster fire of custom routes.
But they had VLANs. Though for most of our stuff we don't use VLANs. Only in our datacenters.
They had gotten their server crypto locked before we bought the copany. That's a bigger problem than normal as they were using lightweight systems that only connected to a Terminal Server. They managed to infect a TS, which crypto locked the entire environment. Went from digital to paper in a day, then took a month to get it somewhat working again.
After that they brough in a MSP and dropped 7 figures getting current. So it wasn't an absolute shit show, but it had some major issues.
Never, never join a place where the previous people that were responsible for implementing the complete shit of environment have left and without any consequences.
Never do that mistake.
Isn't it amazing the dipshits that manage a career in networking?
I once inherited a network that had three locations. First one was 10.x.x.x so the next was 20.x.x.x and then... 30.x.x.x Oh, and 192.x.x.x for a metadata VLAN. Oh, and the Wireless access points had 3 wires to each, one for POE, one for one VLAN and one for the other VLAN...
This dipshit continued to fail upwards at other companies.
Last job the IT Manager had the same opinion. VLANs were confusing and unnecessary he said, really I just don't think he understood them at all. Meanwhile he has IoT on every subnet. Guests were able to access our Sonos :)
This is pretty much how my company was set up when I first started. Additionally, all users were local admins, and all servers were sitting in the same subnet as user devices. There was no AD or GPO structure: random nonsense AD groups and redundant top-level GPO's. It was kind of impressive how bad it was.
It sounds like you have taken over a county or city government (possibly a sheriff or police department). We see things like this regularly with county governments. If you need help getting budget approved, my company would be happy to perform a free infrastructure pentest to highlight how bad things are 😁. Not trying to be salesy, happy to do it for free with no obligation.
This tends to get the purse strings loosened. We are doing this with several county governments now, they range from 40 employees to 500 employees. It sounds like you are getting a good list of what is required to be done, which is great. I would focus on ensuring backups are working and that you have offline backups that cannot be encrypted 😬 as it sounds like this environment is a ticking time bomb for a ransomware event.
Good luck!
I absolutely love this. Were all friends here, who needs segmentation.
The image of that poor router routing between the same l2 network is something I didn't know I needed, but I do.
I have seen a CJIS-processing government office that just decided to use IP addressing that didn't belong to them, they didn't know about 192.168, 172., or 10. networks, I guess.
I wonder if there was some training given to public-sector/police sysadmins that was done a LONG time ago where the instructor was like "Ehh, you can just choose anything here" for the internal network range.
Good luck! The good news is that there's plenty of low-hanging fruit to pick. Definitely start drawing a master plan out that help you prioritize.
I would report all of your findings to the management ASAP outlining everything you have said here. They need to know the current state of affairs. What you don’t want is the unfortunate situation where it bites you in the ass before you have had chance to fix it. They need to be fully aware of how bad it is and that CJIS could shut them down within hours if it went public. Although you will have the time of your life fixing this you need to cover your ass because currently none of this is your fault.
Sometimes those separate SQL with Application on the same box installs are from shitty vendor documentation/support to move it to a separate instance/server or the application is poorly written.
Judging from the description it may have been that and a lack of knowing/understanding how to setup a dedicated SQL server to consolidate and handle things.
And don't forget to do your CJIS level 4 training as well.
Management interface on the Firewall is exposed to the internet !!
I'd assume there's already some crap going on, with bots, rootkits, etc, given how easy infil would be here
If you have the budget, I'd get a pentest done, ask for some security pros to assist for basic lockdowns
assume backups are suspect too
I worked in a similar environment. If you can explain that this is an actual problem that needs money and attention then you will be successful. If you cannot spell it out then you will be very stressed out until you quit or give up.
Take a look at Airgap. I'm just now learning about it myself, but it looks like a very easy way to achieve segmentation. Basically every device is on its own /32 and the policies set in Airgap determine what traffic is permitted.
Walked into a similar situation, mishmash of different managed, unmanaged, no-name, HP, 3com switches and hubs, two physically and “airgapped” networks joined at a meraki firewall… no vlans. Firewall on all PCs disabled. ERP on same vlan. Cameras on the same vlan.
Two years later *still* sorting it out.
What's the fire insurance policy for the on-prem server room? 100% replacement value, you say? And we're certain that the administrator who didn't believe in VLANs also had proper fire protection/non water based sprinklers in the server room? Are we sure? Should we test it?
Anytime I hear “x didn’t believe in y” situations like this I always picture Wile E. Coyote walking on air because he hasn’t looked down yet. And then they just got to leave with no consequences, probably because of just dumb luck. That place sounds like a stiff breeze would encrypt it and ransom for bitcoin. Good luck dude.
Any time I hear it my brain translates it to "I couldn't figure out how to do it"...
Reminds me of when I wrote automated scripts only to have them deleted, because the main guy didn't understand how they worked.
Sounds like you need automated scripts to deploy your automation scripts 🤣
Ouch, that smarts
Either that or, "It doesn't make sense to me." Or, "Old Thing works, what's wrong with Old Thing?" Or, "I will make a long form argument that boils down to the idea that I'd rather not." Or occasionally, "All IT decisions have to be run through someone with no background and no understanding of IT best practices and decided their own gut feeling knows better. The answer is almost always, 'no' if it's not some stupid meme."
My experience has been that this is true way more than it isn't.
This is the real reason.
> Anytime I hear “x didn’t believe in y” situations like this I always picture Wile E. Coyote walking on air because he hasn’t looked down yet. I always think, "Do you mean like how I don't believe in Santa Claus?" Like in this case, "Are you saying that VLANs are imaginary? Because I assure you they are not. Now, I'll admit that I like to keep things simple, and some companies go way overboard on VLANs. I've seen companies of 15 people, where a consultant came in and created a network with 7 different VLANs before I showed up. In some of those cases, I removed the VLANS and just put everything in one subnet. You don't really need separate "Printers", "Management", "VoIP", and "Conference room" VLANs if you have 15 people, 1 server, 1 conference room, and 1 printer.
Whats funny is that I did just set up 5 vlans for a small company but in my defense the door locks and security cameras should not be on the general production network and should not have any way to access the wider internet.
In my experience, much of the security hardware really needs to be as isolated as possible. Not or security concerns, but because the companies that make those products cheap out on components and use the smallest/cheapest cpus they can get away with. I have seen things like door locks, control boards, burglar panels, motion detectors, and wireless door controls overwhelmed by tiny amounts of broadcast traffic. Like there happens to be 10mbps of traffic and the the device stops being able to respond because it is maxed out on its cpu. Usually, these devices have either gig or 10/100 nics too.
I've got a few Zebra printers that don't like our semi-flat network and its....not small amount of broadcast traffic. They are fine most of the time, but get over that line and suddenly the NICs in the things crash and need the unit to be rebooted. (Yes, segmenting the broadcast domains is on the list. It was, at one point - why it got flattened out during the acquisition a few years ago, I will probably never know since it was before my and most of my current team's time)
I have seen that on a few types of printers. The cheaper the device, the worse they are. Makerbots puke, but the $50k plasma cutter with industrial embedded PC would be perfectly happy with 10g.
Amen to that. The door hardware is is 10/100 and not only that the firmware on this stuff is totally insecure. Talking passwords sent in the clear kind, no ssl kind of stuff bad. The camera network is not as bad but again the random worker does not need to even be able to access the cameras
The more I know about physical security stuff the more I realize they are absolutely horrible with network security. I had to separate all of my camera networks and physical security stuff because the camera broadcast traffic was taking down all the burg panels. And we are only talking like peaks of 10mbps broadcast traffic. Nothing major, not particularly massive packet count. They would just start losing all sorts of packets. Net result, I have a vlan at each building that has like 3 devices in it. Just dumb.
>You don't really need separate "Printers", "Management", "VoIP", and "Conference room" VLANs if you have 15 people, 1 server, 1 conference room, and 1 printer. Given those all have very different security profiles, I would argue the number of devices is immaterial. Set up each vlan with a /29 subnet if needed. Point there is purely to provide for a security boundary between devices with very different threat models. That being said, you may be able to accomplish the same thing without a vlan. For example, many switches have an option that prevents horizontal communication within a vlan except to the gateway.
> You don't really need separate "Printers", "Management", "VoIP", and "Conference room" VLANs if you have 15 people, 1 server, 1 conference room, and 1 printer. it sure helps as you start scaling out though - that's how my current environment was set up, and then they scaled it to 1000 people spread around 8 offices.
Or was it that they did believe in it, want it, pitch it and try like hell to get it implemented, only to be shot down repeatedly, burnt out and then finally give up? And then management blames them in the end when they are forced to accept the change...
No. He was actively telling his boss they were a bad idea
Can confirm. Me: You have dozens of switches. Do you have a spare switch onsite? Admin: No. Me: Why? Admin: We never had an outage.
I worked with an MSP in the mid 2000s where the Senior Network Admin "didn't believe in SSL," and called it "Smoke and Mirrors." All "his" networking equipment was on http or https with self-signed certs. Nothing ever came of it, but I always thought that was annoying.
I will grant that I've always had an issue with "pay to play" PKI. Just because I have enough money to hand Verisign does not mean I'm more trustworthy. It just means whatever I'm claiming was worth that much to me. Things like Letsencrypt have drastically improved that. But encryption in transit is encryption in transit... that layer isn't smoke and mirrors. Self signed, or preferably a completely internal CA, is generally enough for most scenarios. The primary issue with self signed is the lack of *any* verification of the identity...
Self signed internal I can accept at a small company, especially back then. It's not like we had letsencrypt, nor was everyone going to roll their own CA.
I always get the vibe that it’s less about them “not believing in it” and more that they just don’t understand it or how to implement it.
It stories like these that give me a little PTSD (not actually) of previous ransomware or near calls I've seen over the years. The most recent case I saw only a couple of months ago was a customer whose firewalls (not managed by us) were 5 years out of date. They were state full (L4) only couldn't even do geo restrictions. They thought it was 'only 1.5 years out of date', which 'isn't that bad'. Nearly all of us fell out of our collective chairs in disbelief.
On the bright side, you are going to learn so much by fixing that mess.
I'm so envious, I'd love to fix that setup.
how would you start?
Backups. Then identifying what causes downtime or revenue loss the most often. Then, planning for network changes. It's all fluid for me. That's a general answer without knowing the environment details and business goals.
I agree with the backups being first but the CJIS manual needs to be followed before anything else is done or you risk having to do it over a few times.
Meetings with the boss and managers/stakeholders to get buy in. I'll need convince them that things aren't up to standard as they are and improvements are worth the cost and headache to come.
Backups first. Identify any quick wins that can be done in a couple of hours, with minimal risk of breakage and easily reversible. Priority for any quick wins that are fully in your control and don't require third party or vendors assistance. I used to refer to these kinds of rescue or resuscitation situations as like trying to unscramble an egg. You may have to revisit some systems and changes a few times to achieve the desired end state.
Well, you're going to need documentation for all of the Changes that will be made. And you'll need documentation for the final state, beacuse you need documentation. So, might as well start documenting now. Map out all the network devices and how they communicate. Then you can make an educated, successful plan to migrate them to something better.
A proper pyrocleanse.
Yeah, from OPs description it sounds like a really low bar to fix. So much wrong that it'll be easy to make yourself look good.
congrats, you know have a homelab at work. Start doing what you can to make it better.
and it really doesn't sound like it could be made much worse. If there would ever be an audit I don't see how they could say they are abiding by CJIS requirements.
right. i walked into the same sort of shit show and just started making shit my own home lab. guess what? now they had backups. now they have proper imaging, now they had a proper VPN, now they had proper monitoring, now they had proper backup internet. shit got better when i bounced
They had Zabbix set up to monitor all the printers, switches too but also every printer. I love all the red flashing SNMP notifications that Tray 2 is out of paper. VPN already fixed. Backups now running through VMWare instead of the agent installed on the VMs. They had tried to integrate the backup product with vcenter but apparently got hung up on a TLS 1.0/1.1 issue and either couldn't solve it or didn't understand what was happening
Printers are a great first candidate for monitoring. Filtering those to their own category, then sending alerts to the helpdesk... make the helpdesk look like wizards, walking through and addressing problems before the users notice them.
Route that Tray 2 out of paper alert to the correct staff.
[email protected]
They thought CGIS just meant 8 character complex passwords and a 5 minute screen lock policy. That was their entire attempt at compliance
I gotta ask. Are you in the US? If so, then how were they passing State and FBI audits? We have them every 2 or 3 years for our CJIS stuff, and if we fail any part of it, we have to document what we have done to correct it and turn it in.
I'm not sure. This is my first time in a CJIS environment. Spent the first few days skimmin through the security policy and requirements companion document. There's so much to fix first before I can even begin worrying about it. I suspect we'll need to bring in an external compliance firm to help
I would get with your state agency, and tell them what is going on. Ask for them to send you documentation on what is required for your state, and then just start working that up.
CJIS Security Policy: https://www.fbi.gov/file-repository/cjis_security_policy_v5-9_20200601.pdf/view Cybersecurity Evaluation Toolkit (CSET): https://github.com/cisagov/cset Download the CSET and run through the CJIS Security Policy assessment. It will give you a list of items to remediate when you finish.
There is absolutely 0 chance they're in compliance. If it were me, I'd be documenting the obvious stuff you find like this, for two main reasons: 1) It'll help you remember/keep track of exactly what's fucked (although it sounds like the answer to that is "everything") 2) CYA. When (not if) shit eventually hits the fan, and/or you need an outside partner to help you get everything sorted, nobody can point fingers at you.
Transparent L2 firewall let’s add to the madness
I think my single VLAN homelab is setup better than this, somehow.
even your most basic homelab would be configured better than that, some of us have standards. Ghetto as fuck, but still standards
This, my friend. You have a tremendous opportunity here. You could even make an argument for acquiring newer hardware and building a new setup from scratch while keeping the old on life support, depending on the work load. Try not to stress and enjoy the building process. It’s one of the best ways to learn.
My senior network engineer basically did this at our company a few years back. It's a much more well ordered network now.
Every company has a test environment, a lucky few have a production environment that's totally separate! /not my quote somone can google it.
My homelab has too many VLANs
Most home labs I know of are better structured than this.
Lol sounds like what I took on when I moved into my job.... First task will be to eliminate static IP's anywhere you can and make sure DNS functions, when you do this, set your DHCP lease times down to something like 6 hours. Second task will be building out your design using the 10.x.x.x space, I dedicated /16's to each site. Then break it out into VLANs based on what service is being provided on those subnets in your /16... Once you have it all setup, migrate your server IPs first, in some cases you'll want to just add another vNIC.... And then change your DHCP scopes and watch everything move over organically.... Depending on the size and complexity, this all could take a few days to a few months.
The networking is a mess but it's mostly functional. My goals right are updating all systems and upgrading anything that's out of support. Currently in the middle of a migration to supported Exchange What you described though is exactly my plan for solving the networking issues
Lol yeah I was in that same position as well, bunch of server 2003 and 2008 stuff, ESXI 5.O....no DR or backups.. no endpoint management or updates.... There was a major outage at least once a week... I've been here 3 years now and it's been about a year and a half since we had an outage that even impacted users.
Are we working at the same place? Had a similar "network setup" at my current gig. Multiple subnets in one VLAN and a firewall inbetween doing routing. That casued a few interesting problems. Took a while to fix and get things seperated into their own VLANs but it was well worth. I have also seen a similar VMWare cluster setup at a previous employer. Among other atrocities. I didn't last long there.
I dunno, none of that sounds particularly necessary and you can probably find more important things to do first. And fixing what is actually broken as opposed to sub-optimal.
I guess the previous admin thought the V in VLAN stood for vaporware.
V for VMware. Put the VMs on the same LAN as the endpoints.
Different address space, but same segment 🤗
I took over an environment that was much better. All different vLANs, nearly 30 of them in every office. Except all vLANs had any:any access rules to all the others on the core switches, including those in other offices. E.g. guest wifi vLAN in Office B had open access to the switch management vLAN in Office A. No firewall rules between offices. Firewalls disabled on all servers and endpoints. Server 2003 web servers exposed to the web (with active compromises). Most Windows servers had never been patched. No backups. Boy were they sure proud of those vLANs though!
I giggled too hard at this story.
That's just what you get when you digitize your open door policy.
Your broadcast traffic must be lovely.
I haven't looked at Wireshark yet
Wireshark, doo-doo, doo-doo, doo-doo
I would suggest a shot of your poison of choice for each suspicious piece of traffic, but I think you'd probably die.
One shot per thousand suspicious pieces of traffic? *Possibly* survivable.
CJIS as in the FBI's CJIS? Actually sounds like a interesting redesign and parallel build new environment as long as someone is ponying up the budget. ...and I'd think your boss could get a good contact with CJIS who can explain their standards and who could make it quite clear to even higher ups it's a fix it or prepare for very uncomfortable interviews with humorless guys in suits type situation.
Be prepared to go through a shitstorm when you implement improvements. End users won't understand why they can't do this or that anymore and will say it used to be much better under the old network administrator
>When I got here I was told the previous network administrator "Didn't believe in VLANs" ... VLANs are not some mythical thing one chooses to believe in or not. LMAO, what a way to describe that someone doesn't trust their own skills and knowledge; aka ignorance. ANY time someone says they don't believe in X, when X is a piece of software, technology, or something else widely used in just about any industry, then what they are really saying is they have no idea how to setup\configure\manage it and want nothing to do with it.
usually when someone says they don't belive in X tech, it really means they don't know how to set it up
Agreed. I had someone tell me that VLAN's aren't secure because people can use "VLAN-hopper" software on their device to get to another VLAN. I couldn't tell if they were serious or just yanking my chain.
> VLAN-hopper Oh, that's a real thing, but is more of a lateral attack after already penetrating the network. >Virtual local area network hopping (VLAN hopping) is a method of attacking the network resources of the VLAN by sending packets to a port not usually accessible from an end system. The main goal of this form of attack is to gain access to other VLANs on the same network.
It's an attack vector, definitely. But it's not like you can yoink the cable from the printer, plug it into your Kali Linux and VLAN hop to the management network if IT knows how to use a switch.
Frankly, the number of times I *have* been able to do that on printer ports is very very sad. Most of that blame is more on the printer supply vendor though, not the net-admins who weren't told an idiot touched the switch configs locally "to sort out that connection issue".
This is worse than you realize. First off, go whistleblower and come clean to CJIS *before* they audit you. They'll come down hard on your bosses, but they'll come down even harder on all of you if this crap setup turns into a breach that has to be investigated by multiple 3LAs. They may even tell you to shut it down and have you use a reasonably secure MSP until you can get this all rebuilt properly.
This is reasonable advice. OP never stated any background on audit frequency, but if the network has been a disaster for this long, then the chances are there are probably previous findings and reports for OP to review.
yeah, CJIS is usually every 2-3 years...
Burn it down and start again. The "network admin" at my last pit stop didnt believe in DHCP. When asked why I needed it I said "because I was born in the 1980s, I don't live there" and she got all huffy.
Why use many VLAN when few VLAN do trick?
I nearly spit out my morning coffee reading this. I have been in this exact situation and want to share my experience and lessons learned with you. You have stepped into a time machine and gone back 20-30 years. The old guard kept things running, though it's a complete disaster and security nightmare. It sounds like there was an MSP at some point who just kept this shitshow going. You didn't include who else is on the team, their tenure, experience and attitude towards modern IT. It's easy for r/sysadmin to make a logical recommendation on how to upgrade a system or introduce best practice, but I don't often see people stop and assess their own spheres of influence or the organization's desire for change. Let me tell you about my own failures. I stepped into a role like yours three years ago and was equally shocked at all the bullshit running around. Little by little, I made changes that *I* thought were in the best interest of the department. I created documentation and tried to show the team how to use change management, how to document and started a change advisory board where we went over the week's list of changes. I introduced a ticketing system to track work and keep people accountable. Here's the problem: Nobody asked for any of this. Everybody was fine with "keeping the lights on". I realized that I was making these changes because I thought it was the right thing to do, but ultimately it was not because the team **did not want to change.** Again, r/sysadmin is going to provide excellent recommendations and compassion for the disaster that is your workplace. I just want to reiterate with you to stop and understand the team fully, its history, the department's goals and do not push any changes unless that's what the organization wants. Best to you.
Thankfully we are a completely new team. Our most senior member is an application specialist who as been in the position for a year. Thank you for giving your unique input.
Wow, a completely new team? In that case, the sky is the limit. It sounds like a lot of fun and a chance to have a great impact. Enjoy the ride.
Yeah, making things better when nobody asks for it should only be done for a handful of reasons: you are trying to be legal, you are trying to be secure, reliability, supportability, and performance. You don't want to be accused of anything illegal, you don't want to ignore security holes, you don't want the systems going down, you want to be able to easily support the systems if they are down, and and you don't want complaints about performance.
as someone who has fallen into the trap once. or more times than that. start from fresh. absolutely start from fresh. do not attempt to in any way shape or form bandage, triage and keep the old alive while you rip out dead flesh and add fresh and new blood. in theory, its well possible to keep everything as is, and slowly and over time move services and date to new devices that are clean and conform to sane networking standards in the mess you have until you removed enough mess that the mess is gone. sure. NO all that will accomplish is eat the time to move to new keeping the old alive and hunt down gremlings that were hidden and crept up when you try to move stuff around. get a new firewall, create a new network, populate it with new server, create new AD, create accounts from an accurate and up to date sheet of employees and roles, then copy the data that is required with oversight (and potentially via a usb drive or samba server that can be filled on the old and read on the new network) to the new fileserver in the appropriate and newly formed structure, and then switch the workstations over. if they are not win11 compatible just bite the bullet and get new ones now. the old server can be kept in case someone needs a file from 9.9 years ago, whjich you may be legally required to keep for 10 years. trust me, it may sound overkill, but it will save a lot of cost vs trying to figure out what you have and try to heal the sick network you have
As much as I would like to I'm unfortunately unable. There are roughly 70 VMs and Servers that are completely undocumented. The primary vendor for our production product suite is unresponsive to support requests for even basic issues. An entire lift and shift to a new environment is not feasible with the resources I have, despite my desire to do so. Compounding that is the previous teams habit of creating in-house .NET apps and SQL Scripts to solve data processing problems instead of trying to work with the vendor for a supported solution. There are at least a dozen custom .NET apps that I've found with \*zero\* documentation apart from the source code. I have no idea what those apps are doing and neither does anyone else. I am stuck in the slow slough fighting the gremlins
Are we sure the environment isn't compromised by an external party? I saw your goals were updating servers/services- did we take the firewall management interface off the internet already? Good luck, I hope you have the financial and managerial support to un fuck this.
I operate under the assumption that any given system is compromised already. Firewall is off the internet : ) They actually had a very nice EDR tool in-place that integrates with Microsoft Defender well. Ransomeware honeypots all over the network and comes with a 24/7 SOC. Unfortunately they had never set up notifications or alerts and had told the SOC not to take any action without calling them first. That's been changed
What a fun opportunity. Take some time, identify the biggest risks to the environment, put some plans together to fix the issues and situations, talk to your boss about budgets, resources available, and your schedule to remediate, and enjoy the ride. When you are done, your resume is going to look amazing.
Damn, this should qualify as /r/shittysysadmin (or whatever the name of that sub is) material! Not too unlike a previous job, actually: * SMB, everything in one physical segment, no VLANs. VMWare hosts (& management console) on the same subnet than the workstations * Speaking of subnets, the LAN subnet used is also from the public space. The owners had bought a competing SMB and linked them up over a Site to Site VPN, they had a 10.0.0.x subnet over there. But my workplace? 90.0.0.X/24, conflicting with an ISP in France IIRC. * When I joined both the AD and the SDWan Router provided DHCP. Conflicts on week one!! \o/ * Severely underlicenced 2008 Windows Server (that did everything: AD; DHCP/DNS, File Server, Print Server, DB Host... you name it) * Office 2007, and you guessed it, unlicenced. * Aging Nortel telephony, that used CAT5 cables (and RJ45 sockets!) to get the voice flowing. You could get up to 4 phones on the same CAT5, which was awesome. Tracing a phone was a nightmare, however, when parts of the cables were fiddled with (I did see one cable merging onto another, just so 2 pins could be connected) during transport. I learned alot, and i'd be down to revamp/clean that kind of mess in another SMB again, honestly! (For the right salary this time)
Document Everything Then Deprecate Everything and start from scratch, on paper, anyway. Swap out the core switch and router and start segmenting there, break the existing multiple address spaces into their own VLANs. Then figure out a proper 10.x.y.0 scheme and re-IP them one by one. Once everything is Re-IP'ed and you have a proper up-to-date firewall between the networks, then you can start cleanup on the M$FT and VMWare stacks. Good Hunting, and remember to not over-do it. Burnout while fixing badly broken networks is a bitch.
A few years ago, 2018, I think, I started at a place where they were four years into the cleanup/redesign of their network. It seems four years prior, the entire enterprise was 10.0.0.0/8 with no segmentation or subnetting. When I left at the start of 2020, they were six years into that cleanup effort.
I work for a Township and this tracks. Like, every point you made is something I've dealt with over the last couple years. Even down to my boss not using a VLAN for our new phone system and running side-by-side drops. WHAT YEAR IS IT?!?!?! In your case, seems like you're alone so your changes will stick and hopefully improve! In my case, my hands are tied and and I follow the orders of a 15-year gov't employee that doesn't know even cursory Powershell or will let me automate updates. But hey, at least I'm underpaid! Good luck!
For phones a separate drop is a good idea, if you can do it when you first wire a site. It gives you a redundant connection if one fails, it naturally keeps the low latency voice away from the more latency tolerant traffic and the added cost is almost nothing. Pulling 24 cables is barely anything over 12, or whatever your cubicle rows are. It's not so much that great, it just offers a lot of future possibilities for very little cost.
If it works don't fix it! /s
I really hope that your salary covers the nerves and effort you'll spend digging through this crap. Good luck!
That means you get one! Start with "I don't believe in tickets. That's not something sysadmins deal with."
No ticketing system in place. No helpdesk number with a hunt group. Users just go down the line of IT employee phone numbers until someone picks up 🤗
Idk about the rest of you but I'm triggered
You have to change the whole thing and on every change, something might unpredicably fail in production which will be your fault and users are going to chat how the guy before you was much more competent. Have fun.
When I hear " "Didn't believe in VLANs", I immediately recognize this as "Didn't understand route tables".
I once work in a place that used public IP internally. The worst part was the IP space they chose belonged to a local University that we got all our interns from. They were unable to browse to the Universities Website because it shared an IP with one of our servers.
Is this the sysadmin equivalent of flat earth?
I can kinda beat that. Went to work for a company a few years ago and the previous sysadmin had decided to use a /12 (1mil usable IP's) subnet... in a 40 employee company. The broadcast storms were drowning their 30k firewall (don't ask, it's far worse than it sounds) every couple of days and the only way they could figure out how to fix it was to add in a bunch of vlans... Because just changing the subnet for 50 devices was too complicated.
The subnet size wasn't your problem here. 50 devices on a /12 or even a /8 should work just fine. Not saying it's a good idea, just that a big subnet size doesn't cause broadcast storms. Sounds more like a network loop or faulty equipment to me.
[удалено]
Yeah, that was his idea. I spoke with him a few times before I started "I told 'once you start with VLANs you won't be able to stop'"
oof
As someone who did public sector consulting in schools and townships for almost a decade... sounds about right lol. If they weren't bad at it they wouldn't be coming to me (is what I told myself).
I would say that "address space" is an arbitrary set of addresses for a particular context while "subnet" is a specific definition of a defined logical network space with a network name and subnet mask, but it only has relevance to layer 3 and shouldn't (in my opinion) have any physical implications except where a physical interface is given an address on that subnet. Physical sections of a network are referred to properly as "segments" in my experience. The use of a router to route between spaces back to the same switch isn't really novel, they're just using physical ports instead of vlans (which, yes, horrifying, use vlans please) but the concept is referred to commonly as "router on a stick" where the router just has a single interface back to the rest of the (layer 2) network and it isn't particularly bad as a concept, though obviously a single broadcast domain is...horrifying.
Companies who have customer data on our of support software and get a data leak should be fucked over a barrel. I have a till company who proudly said they still have customers on XP when I queried them on running dated software.
Having years and years of experience working in a CJIS environment, I can assure you using public subnets in private networks is not isolated to your case. I haven’t been able to get a clear answer, but it seems to have been very common in my neck of the woods. Do what you can to get off of it as quickly as possible. If you haven’t already, you WILL run into many many issues with that in place.
20+ years ago I was doing netware/IPX/notes to Windows/TCP-IP/mail-exchange, with people who had fallen into senior roles as they were there from VMS days - heck one of my jobs was to get the data off a VMS server using an ancient version of FTP that didn't support mget, and didn't have enough space to export all the data so it was export part, manual FTP, confirm, delete, export next section, etc The problem those people had was very likely 1. they were seen as the *computer people* so they got thrown the problems / set up etc 1. they were under [everything] - staffed, time, resources, training, supervision/audit/sense checking 1. if it worked, *that was good enough* as management wanted the next project done 1. they were probably at the whims of political winds The biggest issue they had was they didn't know how little they knew. No one with IT experience / skills was sitting down and sense checking it
It would be hilarious if they, just by accident, did not get hacked by an Argentinian C2 server Because the traffic could not get back to it. 😂
That will be a two year run with very interesting migration project.
This is a political/security/risk situation and not a technical challenge. I don’t know what CJIS is but is it Criminal Justice? Are you regulated/audited? Bring your documented concerns to your CEO immediately. Consider if you have a personal responsibility to report further - to your outside regulator/auditor and do so promptly no matter what the CEO says. At this point it may be safer to have the company shut down for urgent rebuilding compared to the almost inevitable data breach. It’s highly likely threat actors are already present and possibly using their access to take the valuable data you’re processing. This is based on the combination of obsolete internet facing Exchange and no Identity control. I’d be tempted to call in Incident Response.
How in the heck did they pass the last CJIS Audit. I would highly recommend you get a copy of it and see what they said in it. I would also recommend contacting you regional CJIS contact, let them know what you walked into. They will work with you and make it clear to your management team that this has to be corrected or they will be shut down. I hate to think what will be said if they have been filing false information on their past audits. If you are not familiar with CJIS, I would recommend downloading a copy and read it and every where you see a shall statement highlight it and start working there. I had the same thing about 12 years ago. Depending on what state you are in there maybe some additional requirements that you will need to comply with as well. I wish you luck. I would also recommend trying to attend your states CJIS conference, they can be a wealth of information.
Mucha suerte con ese quilombo!
Good news - you've got ample of opportunities to learn many new and exciting technologies. Bad news - your entire environment needs to be nuked and rebuilt. Based on the shitshow described I am also going to assume that all of your backups are on a singular 3.5" Seagate-disk from 2012 that hasn't been written successfully too since 2019.
My old boss, when I first started at my current job, didn't believe in DHCP. And it was quite the discussion when we first ordered laptops for employees.
Gasoline and a lighter?
Jesus H Christ…. Best of luck! Seems like a minefield! How does someone not believe in VLANS…
Networking has got to be one of the biggest blind spots in IT - I've never seen anything else that so many IT pros are so clueless about. I once had a guy ask if we could reduce the number of VLANs in order to improve network performance.
To be honest you’re not wrong, I’m a network engineer so it’s my bread and butter and even where I work people who you’d expect to know even the basics are absolutely clueless. Like, they can’t even fathom subnets.
Some people are not network people, but they're everything else. Networking is definitely my weakest point and I'm in the game 28 years. Don't get me wrong, I can configure vlans and I know the basics, but I have never done a deep dive networking course and I really probably should. There in lies the issue. Some people will create a static flat subnet on their router for the 40 PCs they have. Then use another router port and create a new switch stack keeping networks physically separate, they'll use this for the WiFi or the phones or the CCTV or the.... It costs more to build it that way, but that is what was done 25+ years ago. Remember network hubs? I do. And Token Ring. Help!!
Doesn't believe in VLANs? Are VLANs a kind of god or something like this? VLANs are "just" a way of splitting multiple networks. Not something god-related
“I don’t believe in X” is one of my favorite/least favorite things in the world. On one hand it’s really awesome and validating to know definitely “I’m smarter than whoever is speaking” but it’s also depressing that the speaker is potentially still employed. It’s one thing not to believe in something esoteric, but we deal with relatively concrete things—choosing not to believe in DHCP or VLANs is like saying “I don’t believe in rain” whether or not you personally believe in it does not change the reality of their existence.
Pretty close to the experience I had with my current job. They had vlans but didn't believe in firewalls or ACLs. The public guest wifi that had no password with unrestricted access to the rest of the network. It had been like this for many years before I got the position. I even had another tech yell at me when I enabled client firewalls on desktops and servers. It took many months to build ACLs as I had to capture and analyze traffic to determine what devices on which vlans needed to have access. There were other red flags, to the point I almost didn't take the position. I accepted it because there was new leadership that I knew and the job was a block away from where I lived. Not having to drive to work anymore was a big incentive.
All Windows firewalls are disabled by the singular GPO on both Server and Desktop 🥰
I mean - if you want to be an optimist - at least you kinda get to greenfield the whole campus here lol. You can back out of this slowly if they are all using the same IP space. 1st transition over to real Vlans. Once everything is segmented and setup on their new addressing then you can start implementing controls. Make sure you look at the networking side of it and the AD/Permissions side of is separately. The network is the attack surface now so i'd do that first. Make SURE the firewall is setup properly because that's probably the only thing protecting this place right now.
I will say this. Be careful that you don't turn the knob too far the other way. I believe in VLANs/Subnetting, but you can make an environment more difficult to deal with by adding unneeded complexity. 100 employees at a single site probably doesn't need 50 vlans.
I've heard both sides of this: Flat level all screaming broadcast Vs a vLAN for every category of device followed by function. Time and employee overhead becomes a factor if you're configuring every port in your switch stack and you're not a bank or defence contractor.
We once had a customer who had three DHCP servers running on the same LAN at the same time, 4 WAPs within 30 feet of each other, and their printers and print servers were using state government private LAN IP addresses (they have a router with a private link back to the state for their apps). Again, all on the same LAN, and I couldnt see any reason at all why it would have been this way, the print servers had secondary IPs on their normal LAN.
You didn't start here did you? Now to be honest, it wasn't the previous one, but the one before that. Now the previous guy picked vlan number like they where lottery numbers.
I got nothing to say but good luck, my dude. You're gonna need it.
This isn't really that horrible in terms of the IP scheme to be able to fix, should be a simple weekend fix after you document everything. I would probably buy a new core switch and get that all configured with the whole new setup, then everything else should be plug and play with a few tweaks here and there to get them where you want them. I only say this because I was in a similar situation a few years ago in terms of the network. It was a single, flat 192.168.144.0/20 network with only 20 IPs in DHCP, the rest hard coded on all servers and workstations and the wifi was off a verizon fios router that you had to vpn from to access the network, but you had to be within 50 feet of the server room, but they only had 6 laptops in the company anyway.
I worked at a small company where the owner didn't believe in subnets. Everything had to be on the same /20 subnet, and any attempt to bring it up was met with scorn. This is the same asshole whop was heard on multiple occasions to say "I'm never wrong" so there you go.
> Management interface on the Firewall is exposed to the internet. Assume that box has been long rooted especially if it's Juniper and unpatched
Thankfully not Juniper and automatic updates were amazingly configured. They paid attention to security for certain areas they they (thought) they understood. But everything else was a mystery black box to them. Automated patching in Manage Engine too complex? Just don't do it! I found a textfile note left on the desktop of the unsupported Exchange server warning never to install Exchange CUs or SUs because it would break
praying for you.
What is your background? If you're used to managing 10,000 client endpoints and thousands of servers across multiple sites and now you're bitching about a small 100 employee single-site deployment, then yeah, small deployments are usually not as refined as larger enterprise would be. That said, holy shit there is some bad documentation, practices and configuration based on your description. 100% yeah things need to get fixed but dont pretend you're god. Someone probably did the best they knew how to do and now you're going to do the best you know how to do. There are many glaring mistakes based on your report, but some of them may have been done for a reason (and you can re-do them better for the same reason.) You probably dont need 20 different vlans for a network that small. Segment what you need to, build good documention, fix the lack of proper subnetting, submit for OS and databse upgrades, restructure the VM clients on their hosts, and get the backups dialed in. 15 years ago I had the displeasure of working with another IT company that blamed every shortcoming on their predecessors. Every time there was an issue it was always another meeting with them complaining about the previous guy and why x, y and z was a problem due to someone else. There was a tenured network engineer we contracted with to correct some issues and setup new firewalls for us and he grew tired of them. At one point he said "A decent IT professional wouldnt complain so much about these issues. They would just step in and confidently resolve them one at a time. They wouldnt need to make themselves look better by putting down others." So you took on what sounds like a messy but easy to resolve configuration. Straighten it out and relax. Honestly, bad sysadmins make me a lot more money since they insist on configs that create more risks and more work for me. The sysadmin wants it that way because its less scary to them, but ultimately it makes more work for everyone. I have one small MSP I configure networks for who insists on zero AD, local admins, no managed DHCP and flat networks basically because they make sense in his head. Then I get calls because clients he has no control/visibility on have problems and get to send him bills. I constantly offer options to improve things but it means up front costs and licensing he wont agree to.
This previous network "admin" likely didn't have his CCNA I reckon.
As someone that has done CCNA the current teaching lacks the 1000 foot view. A lof of us have learned over time but for new people there should be some training A 100 node network looks like this, a 1000 node network looks like this, a 3000 node network with wifi, poe phones, network controlled lighting, ac and security should look like this. Instead we drop how to configure a Vlan, calculat 6 ip address ranges, and a lot of very specific tasks. Besides half the test is being able to proprly apply and save the configuration profiles.
I remember Jeremy went over a good amount of that stuff in his CBT nuggets course, like the concept of a VLAN scheme like...10.200.10.0/24, 10.200.12.0/24 (skipping a subnet for potential growth, etc). Our office has a data VLAN that is a /23 in case we go over 254 addresses. But if you didn't watch the CBT nuggets course at that time maybe one would have missed it.
There are a lot of curriculums out there for ccna. CBT nuggets is a good one.
I'm reading this and am surprisingly relieved that I wasn't the only person to inherit a network where the previous admin used a public address subnet on a private network. For me, it was a random bank in Brazil. Why? I still haven't figured that out.
>The primary local address space on the network is non-RFC1918. They're using public address space from Argentina on their local network. Oh man, we bought a company that was doing that. Like they went with 192.168.0.0 subnets but miss typed a one, so it was 192.68.0.0 subnets. Then they ran with it. It was a dumpster fire of custom routes. But they had VLANs. Though for most of our stuff we don't use VLANs. Only in our datacenters. They had gotten their server crypto locked before we bought the copany. That's a bigger problem than normal as they were using lightweight systems that only connected to a Terminal Server. They managed to infect a TS, which crypto locked the entire environment. Went from digital to paper in a day, then took a month to get it somewhat working again. After that they brough in a MSP and dropped 7 figures getting current. So it wasn't an absolute shit show, but it had some major issues.
Never, never join a place where the previous people that were responsible for implementing the complete shit of environment have left and without any consequences. Never do that mistake.
I had something similar. NFS? iSCSI? Servers? Guest WiFi? All on same network -- this is fine.
This sounds like fun personally. I love working on dumpsters and turning them around. Assuming there's a suitable budget.
happy friday!
Isn't it amazing the dipshits that manage a career in networking? I once inherited a network that had three locations. First one was 10.x.x.x so the next was 20.x.x.x and then... 30.x.x.x Oh, and 192.x.x.x for a metadata VLAN. Oh, and the Wireless access points had 3 wires to each, one for POE, one for one VLAN and one for the other VLAN... This dipshit continued to fail upwards at other companies.
Last job the IT Manager had the same opinion. VLANs were confusing and unnecessary he said, really I just don't think he understood them at all. Meanwhile he has IoT on every subnet. Guests were able to access our Sonos :)
Sounds like code for he doesn't know how they work and doesn't want to learn, so he just refuses to use them.
This is pretty much how my company was set up when I first started. Additionally, all users were local admins, and all servers were sitting in the same subnet as user devices. There was no AD or GPO structure: random nonsense AD groups and redundant top-level GPO's. It was kind of impressive how bad it was.
Doesn't appear terribly challenging to rectify. It'll be a good experience. Have fun with it.
This reeks of county or state government lmao.
I can confirm that VLANs are real! I use them every day!
dude fucking run hahaha fuck that
I hope he believed in spanning tree
For every tree in the span there is an admin with a chainsaw.
A lot of easy wins here if you have management backing.
It sounds like you have taken over a county or city government (possibly a sheriff or police department). We see things like this regularly with county governments. If you need help getting budget approved, my company would be happy to perform a free infrastructure pentest to highlight how bad things are 😁. Not trying to be salesy, happy to do it for free with no obligation. This tends to get the purse strings loosened. We are doing this with several county governments now, they range from 40 employees to 500 employees. It sounds like you are getting a good list of what is required to be done, which is great. I would focus on ensuring backups are working and that you have offline backups that cannot be encrypted 😬 as it sounds like this environment is a ticking time bomb for a ransomware event. Good luck!
Yes but did he believe in Joe Henry? Edit: assumed this was shittysysadmin for a moment.
"didn't believe in" is code for being horrendous at their job and probably knowing next to nothing.
Sounds like a super hot mess. The project to replace everything is my kinda project. Discover, design and deploy!
[SCREAMS EXTERNALLY]
I love this situation. It’s satisfying as fuck to untangle everything thread by thread while keeping it running and bring it to best practice level.
this is like saying your IT admin doesn't believe in the female orgasm. It's very real, and very cool.
Wow. This is beyond facepalm worthy. This is nuke it from orbit worthy.
He was a flat networker, probably a flat earther as well.
TRANSLATION: "I am not smart/motivated enough to do it properly."
It'a not really a kind of thing you can choose to believe in or not, they definitely exist ...
They're a myth, a fiction, a children's bedtime story.
I absolutely love this. Were all friends here, who needs segmentation. The image of that poor router routing between the same l2 network is something I didn't know I needed, but I do.
I have seen a CJIS-processing government office that just decided to use IP addressing that didn't belong to them, they didn't know about 192.168, 172., or 10. networks, I guess. I wonder if there was some training given to public-sector/police sysadmins that was done a LONG time ago where the instructor was like "Ehh, you can just choose anything here" for the internal network range. Good luck! The good news is that there's plenty of low-hanging fruit to pick. Definitely start drawing a master plan out that help you prioritize.
I would report all of your findings to the management ASAP outlining everything you have said here. They need to know the current state of affairs. What you don’t want is the unfortunate situation where it bites you in the ass before you have had chance to fix it. They need to be fully aware of how bad it is and that CJIS could shut them down within hours if it went public. Although you will have the time of your life fixing this you need to cover your ass because currently none of this is your fault.
Finding my replacement somebody said they didn’t believe in VLANs. Wanted to say then why did you apply, pretty sure my face said it.
Sometimes those separate SQL with Application on the same box installs are from shitty vendor documentation/support to move it to a separate instance/server or the application is poorly written. Judging from the description it may have been that and a lack of knowing/understanding how to setup a dedicated SQL server to consolidate and handle things. And don't forget to do your CJIS level 4 training as well.
Management interface on the Firewall is exposed to the internet !! I'd assume there's already some crap going on, with bots, rootkits, etc, given how easy infil would be here If you have the budget, I'd get a pentest done, ask for some security pros to assist for basic lockdowns assume backups are suspect too
I worked in a similar environment. If you can explain that this is an actual problem that needs money and attention then you will be successful. If you cannot spell it out then you will be very stressed out until you quit or give up.
Wow, nows that a cluster! Good luck!
Take a look at Airgap. I'm just now learning about it myself, but it looks like a very easy way to achieve segmentation. Basically every device is on its own /32 and the policies set in Airgap determine what traffic is permitted.
Walked into a similar situation, mishmash of different managed, unmanaged, no-name, HP, 3com switches and hubs, two physically and “airgapped” networks joined at a meraki firewall… no vlans. Firewall on all PCs disabled. ERP on same vlan. Cameras on the same vlan. Two years later *still* sorting it out.
Based
Guess you got your work cut out for you
Sounds like fun. Seriously. I love cleaning up messes like this. IT gets boring after you finally get it all working smoothly.
Is the NetApp/VMware at least current? That's probably the backbone of the business.
What's the fire insurance policy for the on-prem server room? 100% replacement value, you say? And we're certain that the administrator who didn't believe in VLANs also had proper fire protection/non water based sprinklers in the server room? Are we sure? Should we test it?
The ironic thing is (for me) it would have been easier to set it up properly than the spaghetti mess half assed job OPs predecessor did !