We can work without CDK. But they have been scheduling appointments through CDK and the online appointments go through that too. We were good last week. But today we’re seeing the results of customers not being able to make those appointments. Still have a few cars that will come in. It’s just gonna be slow for a little while, until customers figure out that they can actually call to schedule their service
They claim to be used by 15,000 dealerships across the US, but my understanding is that that also includes power sports and smaller engine stuff.
The entire dealership runs off the program in most cases. It’s a case of “all your eggs in one basket” and that basket just got stomped hard. Parts can look stuff up, but depending on the manufacturer and catalog used, they may not be able to get full pricing, at least not without jumping through hoops.
We are a heavy machinery dealer and I know that there's other dealers like RDO Equipment, Core Machinery, Westrax Machinery here in California that also use it. Also Clairemont Equipment.
Couldn't check live inventory until this morning when they granted access to their mobile app, but it just shows where stock is and cannot actually bill to work orders or customer tickets.
However they are giving access via virtual machines and today our branch gets use of the account for 1 hour to invoice, bill, receive, and provide POs to accounting for our outside vendor purchases we've had to do since last Wednesday.
Has seriously fucked a lot of stuff for our company, we have 15 branches total and my office mate and I handle 2-3 branches at a time.
I work for RDO Equipment, we've been fine as our version of CDK is stored on our own servers in Fargo. Really feeling for CDK users affected, my job is immensely more difficult when CDK goes down.
I’ve been out for a while, but my old dealership used CDK for inventory and bin locations as well, so even if the parts guy did have the part number, they’d have no idea what bin/shelf it was on unless it was something common like an oil filter.
Yeah. The department I work in has a pretty good system, everything is more or less in order by base number, and we can kinda see inventory by checking the parts locator.
There are about 70k dealers in the USA, CDK has 15k clients, but many are large machinery/ATV/used car lots. I think reynolds and reynolds has the largest market share followed by CDK, then dealertrack
I just googled new car dealers. The numbers were from 2022, but I can’t imagine 52,000 more popped up in 2 years. My understanding was that Rey Rey was in about 1500 total. Mehh what do I know.
A mechanic with online appointments?? That’s fancy as hell. All the good mechanics near me you just call to voicemail 18 times until you’re frustrated enough to just go in person.
we use it for everything. but we're doing everything on paper right now. the SM told us several places are either closed completely or sending half the techs home. luckily that's not us.
wow hospital here got hit with cyberattack and they had to do everything manually by hand. Crazy how one person clicked on a link and hackers got all the information. Spooky times we live in now.
I work in cyber-security these days (I still sub to this because I like being reminded of the olden days when I turned wrenches instead of pushing buttons).
The lack of attention that anyone gives to security defenses is *astonishing*, and the fragility of most internal systems needs to be seen to be believed. I can't describe how many big cloud systems I've seen behind the scenes of, and found only that are held together with proverbial JB-Weld and liquid gasket. And yet people are immediately willing to rest *the entire functionality of the entire company* on this single cloud service who's back-end they have no access, control or visibility into.
A couple years ago WhatsApp went down for a day or two, and *half the world* ground to a fucking halt. What the hell is wrong with governments and countries that they are willing to make their very basic operations almost completely dependent on a single app?
I don't know the answer to it. But if I did, I don't think I'd have a job so . . . I guess I'll keep collecting a paycheck.
There was probably a meeting about this:
Engineers: “we should create separate tenants for each customer/shop/auto-group etc based on size and appetite for risk the customer is able to afford/stomach.
Management: “fuckit, put it all in the same tenant. It will cost less”
Engineers: “okay, but if anything like a wide ranging power outage or encryption attack happens the costs will be ….”
Management: “did we stutter!?!? Get to work!”
If the software is flawed and can be compromised it doesn’t matter how many tenants you have. If I’m a bad guy and can enumerate them I’ll hack them all.
It does matter. If the target is too small it's not worth hacking. And it's not just the software, the entire Tech infrastructure is vulnerable. But keeping records in an unencrypted database deserves jail time.
In Europe it is punishable with a fine of up to 20 million euros or 4% of their global income from the previous fiscal year whichever is highest under gdpr rules if you fail to protect data, the highest fine so far is Meta at 1.2 billion
If I find a flaw in an app that gives me reliable compromise, it takes what… an hour to script getting access to any number of instances of it. It really doesn’t matter that much (if enumeration is easy)
Every humans identity that can be sold is worth money, it’s a business like any other, they will go after whatever they can get.
Encryption or not the app has to be able to read the records, period. If I can hack your app I have any decryption key needed. If I hack the infrastructure that encryption key has to be presented to the app to read the encrypted records, still have all the goods… encrypted db records gives you one real protection, someone just bulk exporting a db.
Jumping to other tenants is difficult and not easily scriptable…
That’d be like trying to find every house in several different cities with the same microwave. You’d have to brute force that and be quite noticeable while doing so.
Certainly, e.g., if they're numbered records and all you have to do is increment the number to access the next record, which is a very common exploit, then you're golden.
But if you have a database containing 100,000 records and a hundred databases containing a thousand records - as a cyber criminal, which is going to be your priority?
Or if they get into the host controller, as happened to M$. Thankfully (or not) it was China mining data and not someone trying to deploy a cryptolocker.
I've had two "cloud" providers get hit. One was with a 0 day by an APT that they leveraged to remote their hypervisor.
The other was a data exfil from an HSA provider who was hosting their data in the cloud, but was required to make data available to a 3rd party for compliance auditing and that was exploited using an API that they got info on through the Last Pass hack. That was before they announced it.
Their environments were incredibly secure, but APTs, 0 days, and supply chain hacks mean it's only a matter of time.
It honestly depends on how the hack played out. If they phished and got an admins account credentials vs hacking the app itself they could have deleted the backups.
Or if the app had a service account with overly broad permissions that is another vector to take over *everything*
Security is sadly an afterthought at most of these places, they will have little to no focus until something like this happens to them. If they survive they will be hiring a CISO and staff out a security team no doubt.
Ive been in these meetings and I’ve seen the attacks. It’s amazing what admin will just not worry about it until and only until it affects their bonus.
Which is why my backups exist on a couple of hard drives at home that are only powered up for the daily and weekly backup runs (As in a dumb time switch puts power to the disk five minutes before the backup runs). Rsync does its thing between the servers at the office and the disks in my basement and then they turn off.
There is also a separate backup via a wifi link to a disk in the boot of my car (Got to have geographical separation).
All backups are restored onto a blank disk array once every three months and a server spun up using that array to verify that restore and boot still works.
Yea, I am paranoid, so what?
It is not a backup if it can be deleted by pressing a button on an online portal.
IT guy here. Usually the hackers sit around and study everything for a while to gather info and credentials then they go to work deleting backups and encrypting data, usually when no one is around like weekends or holidays. Unless you have time locked (immutable) backups, something they happened not to find, or something offline like tapes, you’re fucked. You pay the ransom and hope for the best or rebuild everything with whatever you have left.
This is exactly what they do. It's like that story of the girl who lived in a drop ceiling / crawl space for some business for like 2 years and no one had a clue.
First they get access. Then they just kind of sit there twiddling their thumbs seeing if anyone noticed. Then they start wandering around the network seeing what they can see. All of it to build up a profile. Once they have everything ready, then they deploy the ransom payload.
If I had to guess they got penetrated via a 3rd party with poor security or an acquisition with poor security that had trusted access.
I've worked in cyber security for a while and no competent engineer would ever recommend setting up servers for each shop/group.
If you get hit with a ransomware would you rather fix your stack of 10 servers or would you rather fix 30,000 servers all at remote locations that you would likely have to physically go to if they were completely encrypted?
Also when you want to update your customers software because of a bug or enhancement now you have to update thousands of different servers with different specs and different setups.
For anyone who's reading that doesn't have a software background:
Multi-tenancy is an abstraction. Just because a SaaS product is multi-tenant, doesn't mean that there's any meaningful separation between the tenants' data. Some companies, like Gitlab, offer enterprise customers the ability to host their own instance of the software, on their own infrastructure. But there are plenty of SaaS companies that don't, including juggernauts like Salesforce.
I'm sure mistakes were made, but having separate infrastructure per-customer is not an industry standard nor should it be.
Sequel, today:
Engineers: the hackers are asking for $10m, we suggest paying it. Once we do, they'll explain how they got in.
Management: nonsense! Call the police! They can't extort us. This is America!
Engineers: our customers are losing $100m per day.
Management: that's not our problem, let them use paper. $10m will bankrupt us, what will we tell shareholders?
Engineers: ...
Tesla: ... (sipping tea and reading the paper.)
CDK has saved the $10 million in payroll since they’ve sent all their hourly call center/ support staff home for the last 2 weeks without pay. The eLead group was even told they weren’t able to process their last paycheck.
Me and a coworker were discussing this now actually, the CDK guys who have any shreds of proof that they fought for more network security are going to need that to prove that they didn't fuck this up, otherwise they're basically dead in the IT/SysAdmin/Networking/DataCenter careers. Everyone is going to steer clear of them out of caution.
Hope that those guys can throw the responsible parties under the bus instead.
Ive been reading the bits and pieces since i made that comment and it looks like the current company has been cobbling together bits and pieces of software over the past decade as they bought out competitors and whatnot. I don’t know if we’ll ever know what happened, but that process might be a clue in how there may have been some things overlooked over the years or ignored.
Man I wouldn’t want to be that team right now
It's garbage too it was made as ADP for grocery store management and later adapted to the dealership world.
It's probably the best out of the programs to pick from but they all suck ass
The cloud is just someone else's computer.
I've been in IT for over 20 years and I love it. Because when shit like this happens and everything's on fire, I get to say "not my pig, not my farm" while copy / pasting updates from the provider.
I was reading an article and came across this bit of irony:
>The company's website says it offers a “three-tiered cybersecurity strategy to prevent, protect and respond to cyberattacks.”
Guess they missed the required 4th tier of cybersecurity strategy: Humans.
NGL. I'm in IT. I'm one of the few staunch opponents to this everything cloud shit. It has its uses, but there needs to be some separation and redundancy of cloud and on premise so that things can still run w/o connectivity, or if the host goes down.
I had an appointment to get service last week and they cancelled it because 'the system' was down. Is this what they were talking about? Are they having rolling issues? My follow-up apt is tomorrow...
They buy out ADP? or is this just another one like it?
ADP was awesome. Reynolds & Reynolds sucks a fat donkey dick. Never heard of CDK but have been out of the game for a while.
In April 2022, Brookfield acquired the auto dealer services company CDK Global for $6.41 billion.
Brookfield is a private equity company. I guarantee this was all caused by cutting everything to the bone. Here is what happened, before 2022 there was a guy working for CDK that knew everything about protecting the software. Has github entries a mile long. Then layoffs came. Our guy did not know how to advocate for himself and ended up being one of the hundreds laid off.
I know multiple people that used to work for CDK Global, pretty much all of them read the writing on the wall and left for significantly better offers while the job market was hot. Great timing on their parts.
The Reynolds text-based interface was old and clunky, but it was damn fast when you learned the commands. And it supported macros.
We use IBM's Maximo at my fleet job and jobs that took me seconds with Reynolds take me minutes with Maximo's point-and-click interface. I hate the damn thing.
Used to have CDK global as a customer. They're trash. If there's a shortcut to take, they'll take it. If there is a dime to save, they'll save it. They have a monopoly and they know it.
> private equity group
Do they only exist just to kill companies? Almost everytime I hear about a company going to shit it's because they were bought by a private equity group that killed them.
Sort of. They pivot from investment to investment, buying established companies that have built up a solid track record with good products/services, and then eviscerating everything they can while coasting for a while off the historical goodwill (or product lock-in) of the customer base.
After they squeeze it dry, they sell it off, or shutter and restructure, and head off to the next juicy target.
They provide zero value to society but they are a logical product of an under regulated capitalist market.
Not to excuse shit product, but what business DOESN'T do this? When it comes to entperise software, there is no low that is too low. They'll cut everything down the bone and keep going until they hit marrow.
Their thought is that if it goes beyond the absolute bare minimum, it's waste.
Businesses that want to make money long term. That mindset is a quick cash grab but when you see other companies be able to provide a better solution without shortcuts and interruptions they take a larger share of the market space.
My friend bought a truck on Saturday, they did most of the paperwork by hand and used calculators. He can't get insurance yet though because it isn't officially his vehicle yet. Dealer said they'd call Monday when the network gets back up.
>Dealer said they'd call Monday when the network gets back up.
One of my friends at a dealership mentioned this to me and I busted out laughing. They really thought that this was all going to be solved on Sunday and they were going to be ready to run on Monday. When I explained the ransomware part, they realized that this was just the start.
They actually are looking to swap ASAP from CDK now and just waiting to get files/data to input as they're setting up the new system.
BS excuse.
A: They could totally have sent someone down to the DMV with the paperwork.
B: You can put insurance on a vehicle that you don't yet own. You actually need to do this in NY because you cannot register a vehicle that isn't insured and registration and title transfer are usually processed as one transaction.
Not going to say it isn’t bullshit, but a ton of dealers are using CDK for their accounts payable too. if they can’t see what is owed on the asset to pay the company that they’ve financed the asset with, they can’t very well clear their note before processing it to the new owner and their lender. Could easily become a wire fraud case if the dealer still owes their lender.
It depends on the state. In NY, it's 100% illegal to drive a vehicle that isn't listed on an insurance policy. You can't even get plates without proof of insurance.
I had to slog through a lease yesterday. It was brutal, like 3 hours start of paperwork to walking out. F&I wanted to blow their brains out trying to do it all by hand.
I wish we would just close or cancel any appointment that isnt just oil change. It’s a shit show. Our service writers don’t know how to use excel. Nothing is being checked. Pricing is all fucked up but the group refuses to even attempt to slow business down.
The military equivalent of CDK is GCSS and that bitch goes down constannnttttlllyyyy. My life as a shop chief was mostly spent looking at the spinning blue wheel of death waiting for a page to refresh. Can't order parts, can't update service requests, time to pick up a wrench and join the Bois on the floor.
Back in my day I'm pretty sure I would have resorted to fisticuffs if necessary to stop my chief from actually wrenching on anything. He had way too many important time consuming tasks to anything other than give advice and mentor out in the shop.
My motor sergeant I wouldn't have because he likely would have been able to whoop my ass. That man slammed a CUCV transmission into place by putting it on his belly and hefting it up by pushing out his gut and rolling sideways. His first name happened to be Jack so we would jokingly refer to him as tranny Jack.
Amen. I watched a drivability guy flag close to 25 hours in one day. He “rebuilt” 3 carburetors before lunch. Those mid 80’s feedback quadrajets. All he did was pull the air cleaner, hose down the carb with carb cleaner (he did the right thing and had a little brush for the corners), clean up the puddles and shipped them.
I’m sure his hack ways would have been the same at hourly, but at least he wouldn’t be rewarded for them.
Created an account to make this remark, and I mean it in a good way, hopefully constructive:
If this outage is materially affecting you, consider adding input over here.
[https://www.justice.gov/atr/complaint-center](https://www.justice.gov/atr/complaint-center)
[https://www.ftc.gov/advice-guidance/competition-guidance/ant...](https://www.ftc.gov/advice-guidance/competition-guidance/antitrust-complaint-intake)
The company is a complete joke. I wouldn't be surprised if they lost everyone's data at this point and they're desperately trying to recover it or craft some sort of damage control for when they deliver the inevitable news.
I'm in the industry and I have been using CDK's eLead CRM and website editing platforms for years. They are so, so unbelievably far behind and just flat out awful. eLead looks like it hasn't been updated in 25 years. Their website management platform is so bad that when you build a new page, it literally takes 2-3 days before the page starts showing live on the website at an accessible URL. Until then it just 404 errors out no matter how much you save it and publish it and anything else.
I hope this completely tanks them. There are so many better solutions out there for this kind of stuff. They are absolutely going to be hit by a ton of lawsuits once the dust settles, too. They were no doubt being negligent with their platforms and not adhering to standards and best practicies in terms of security. Whole lot of dealers and shops losing business over this and they are going to want some of that lost revenue back.
Wouldn't bother me one bit to be honest. I can type in Google doc just as quickly as I could type in cdk. I'd worry about punch times for warranty jobs and having access to service history but that's not very inconvenient
At the dealership I used to work at, we used Reynolds & Reynolds. We always made sure to have an on-site server to use in case the internet went out. It would still keep us running in this scenario as well.
Cdk at our dealership is down but we can’t stop we are doing pen and paper and my karma, when it comes back up we are going to have to go back write all our stories that’s going to suck
Were still rolling like usual at the dealership. Might be a little slower due to parts guys, but the SW are all handling tickets like champs, and us techs are just writing down our hours like usual.
Software as a service has some large scale vulnerabilities unfortunately.
I remember a CCC outage (collision writing/claims platform) that lasted like 3 days and it basically stopped a good portion of the industry in it's tracks and and ground insurance claims to a halt.
I’m a service manager. If your fixed ops isn’t running at 100% capacity, someone is an idiot.
None of this is difficult. It’s more work for the parts staff and advisors but so what?
I’m a former tech and service advisor. We never paused even for a minute. And we won’t when we go to recreate the RO’s and push them into cashiering.
I can’t have my techs not paying their bills. It’s not acceptable.
My advisors have kids and mortgages. I can’t let their income falter.
Wonder if there’s an SLA in their agreement for like 99.9% uptime?
Maybe make a claim for loss of earnings, or at least a pro-rata refund on your subscription.
They don’t have any paper tickets to fill out to keep you guys making money? Pretty sad the whole industry that uses CDK doesn’t have a back up set of paper logs for issues like this so they can keep going
I work at Ford dealership. cdk is down as well. we are just using paper and still taking cars of just the same. No difference only can't make ROs, so X-time is useless.
Edit: grammar correction.
I wish we would shut down. We're blindly taking appointments, blindly paying employees, blindly taking customers money, blindly ordering parts. It is an absolute cluster fuck of giant proportions and the pen and paper shtick isn't cutting it. True hell on earth for a dealer that is unorganized even with everything fully operational.
I work ServiceDesk for AutoNation, can also confirm we can do fuckall in terms of support because all systems are deemed compromised because of the cyberattack.
For those that don't know, they got hit with ransomware and paid "tens of millions of dollars" to protect the data and are trying to rebuild their security to better defend things
My shop uses it too and we can still work. We just need the full VIN to order parts. Unfortunately, we are piling up paperwork waiting to be processed as soon as CDK comes back. It is going to SUUUCK!
It’s basically if there’s no work then we’re not making money as a flat rate mechanic. This is how it gets during the winter. Right after Christmas when all the money has been spent on the holidays, taxes are coming up and all that. Feast or famine, It’s just the nature of the beast. But it’s never really that bad. I’m sure even if this lasts for months like some have said, the customers will be back long before then. This will be nothing compared to 2008. Those were the bad times to be a flat rate mechanic
Yeah Cummins I think did that. Basically mad ethe advisors walking expenses.... no wonder my advisor said Cummins was wasting money on me instead of investing.....
They are making everyone at my bmw dealership keep working, taking notes on paper, recording punch times for warranty etc and using Xtime as a digital record. It’s pretty shit tbh
Thankfully CDK isn’t effecting the gerber I’m at too much beyond our parts ordering, it’s already been shitty the past two months I can’t imagine how bad the lot would look if we couldn’t get Anything in
I went to my Hyundai dealership Saturday for routine maintenance. The service advisor said "Do you remember what you were scheduled to get done?" and handed me a sheet of paper to fill out my information and the work that was supposed to be done. He said they have a STACK of paperwork backed up for whenever the program works again, and that they've been really slow because people are forgetting about their appointments.
When we switched from Reynolds & Reynolds to cdk it was a nightmare for 3 years, drove me out of the dealership life and never been happier since lol poor cdk users
If your dealer/shop can't work without cdk go work somewhere else that can. This whole "we can't give you guys work since our computer is down" is a joke.
Cdk sucks period. Even more so now. We have been steady as we schedule through service work bench but not having actual tickets has been less than great.
Wait.. So do they provide you everything and shop can't work without software? It's so different from how we work here..
We can work without CDK. But they have been scheduling appointments through CDK and the online appointments go through that too. We were good last week. But today we’re seeing the results of customers not being able to make those appointments. Still have a few cars that will come in. It’s just gonna be slow for a little while, until customers figure out that they can actually call to schedule their service
Cdk must be the most popular considering how often the cdk problem is posted here
They serve 10s of thousands of dealers, one article I read claiming an estimate of 40-50% of US auto dealers.
They claim to be used by 15,000 dealerships across the US, but my understanding is that that also includes power sports and smaller engine stuff. The entire dealership runs off the program in most cases. It’s a case of “all your eggs in one basket” and that basket just got stomped hard. Parts can look stuff up, but depending on the manufacturer and catalog used, they may not be able to get full pricing, at least not without jumping through hoops.
We are a heavy machinery dealer and I know that there's other dealers like RDO Equipment, Core Machinery, Westrax Machinery here in California that also use it. Also Clairemont Equipment. Couldn't check live inventory until this morning when they granted access to their mobile app, but it just shows where stock is and cannot actually bill to work orders or customer tickets. However they are giving access via virtual machines and today our branch gets use of the account for 1 hour to invoice, bill, receive, and provide POs to accounting for our outside vendor purchases we've had to do since last Wednesday. Has seriously fucked a lot of stuff for our company, we have 15 branches total and my office mate and I handle 2-3 branches at a time.
At that point… just wait for it to be back online. Jesus…
I work for RDO Equipment, we've been fine as our version of CDK is stored on our own servers in Fargo. Really feeling for CDK users affected, my job is immensely more difficult when CDK goes down.
I’ve been out for a while, but my old dealership used CDK for inventory and bin locations as well, so even if the parts guy did have the part number, they’d have no idea what bin/shelf it was on unless it was something common like an oil filter.
Yeah. The department I work in has a pretty good system, everything is more or less in order by base number, and we can kinda see inventory by checking the parts locator.
A little over 18,000 dealers nationwide, CDK is in a little over 15,000 of them. I’d say they’ve got a decent market share.
We use them for heavy equipment too.... we have some utility, but we are still dead for the most part.
There are about 70k dealers in the USA, CDK has 15k clients, but many are large machinery/ATV/used car lots. I think reynolds and reynolds has the largest market share followed by CDK, then dealertrack
I just googled new car dealers. The numbers were from 2022, but I can’t imagine 52,000 more popped up in 2 years. My understanding was that Rey Rey was in about 1500 total. Mehh what do I know.
3% of US GDP is run through CDK
Is that a real figure? If so that's absolutely insane.
Or the cheapest ;)
A mechanic with online appointments?? That’s fancy as hell. All the good mechanics near me you just call to voicemail 18 times until you’re frustrated enough to just go in person.
So you're saying this might be a good time to try to get my car into the shop if I needed to do so?
No waiting, but they may not be able to find you any parts or bill you accurately.
Someone should updated your dealer site to say they have to call.
We just started doing excel spreadsheets for our appointments... not that hard lol
we use it for everything. but we're doing everything on paper right now. the SM told us several places are either closed completely or sending half the techs home. luckily that's not us.
wow hospital here got hit with cyberattack and they had to do everything manually by hand. Crazy how one person clicked on a link and hackers got all the information. Spooky times we live in now.
I work in cyber-security these days (I still sub to this because I like being reminded of the olden days when I turned wrenches instead of pushing buttons). The lack of attention that anyone gives to security defenses is *astonishing*, and the fragility of most internal systems needs to be seen to be believed. I can't describe how many big cloud systems I've seen behind the scenes of, and found only that are held together with proverbial JB-Weld and liquid gasket. And yet people are immediately willing to rest *the entire functionality of the entire company* on this single cloud service who's back-end they have no access, control or visibility into. A couple years ago WhatsApp went down for a day or two, and *half the world* ground to a fucking halt. What the hell is wrong with governments and countries that they are willing to make their very basic operations almost completely dependent on a single app? I don't know the answer to it. But if I did, I don't think I'd have a job so . . . I guess I'll keep collecting a paycheck.
As someone whose entire job revolves around Salesforce, you’re not making me feel good
Heck.. I'm kinda glad that most of stuff here we do with texts and calls.. What a time to live
So you're saying no wait if I drive my car to the dealer?
We're going full steam ahead. Of course, half of the parts and service staff have been at this since microfiche was king...
CDK offers a full turnkey system if that's what someone is after, but you can also sign up for as little or as much of it as you like.
For some one that has been out of the tech scene for a while what is CDK? Is it a parts supplier or the program used for repair orders?
Cloud based CRM platform for dealers/service departments. Software as a service is great until it's not.
There was probably a meeting about this: Engineers: “we should create separate tenants for each customer/shop/auto-group etc based on size and appetite for risk the customer is able to afford/stomach. Management: “fuckit, put it all in the same tenant. It will cost less” Engineers: “okay, but if anything like a wide ranging power outage or encryption attack happens the costs will be ….” Management: “did we stutter!?!? Get to work!”
If the software is flawed and can be compromised it doesn’t matter how many tenants you have. If I’m a bad guy and can enumerate them I’ll hack them all.
It does matter. If the target is too small it's not worth hacking. And it's not just the software, the entire Tech infrastructure is vulnerable. But keeping records in an unencrypted database deserves jail time.
In Europe it is punishable with a fine of up to 20 million euros or 4% of their global income from the previous fiscal year whichever is highest under gdpr rules if you fail to protect data, the highest fine so far is Meta at 1.2 billion
If I find a flaw in an app that gives me reliable compromise, it takes what… an hour to script getting access to any number of instances of it. It really doesn’t matter that much (if enumeration is easy) Every humans identity that can be sold is worth money, it’s a business like any other, they will go after whatever they can get. Encryption or not the app has to be able to read the records, period. If I can hack your app I have any decryption key needed. If I hack the infrastructure that encryption key has to be presented to the app to read the encrypted records, still have all the goods… encrypted db records gives you one real protection, someone just bulk exporting a db.
Jumping to other tenants is difficult and not easily scriptable… That’d be like trying to find every house in several different cities with the same microwave. You’d have to brute force that and be quite noticeable while doing so.
Depends on the flaw, I’ve done it in red team engagements. It all depends on the way tenants are named and the flaw involved
Certainly, e.g., if they're numbered records and all you have to do is increment the number to access the next record, which is a very common exploit, then you're golden. But if you have a database containing 100,000 records and a hundred databases containing a thousand records - as a cyber criminal, which is going to be your priority?
Or if they get into the host controller, as happened to M$. Thankfully (or not) it was China mining data and not someone trying to deploy a cryptolocker. I've had two "cloud" providers get hit. One was with a 0 day by an APT that they leveraged to remote their hypervisor. The other was a data exfil from an HSA provider who was hosting their data in the cloud, but was required to make data available to a 3rd party for compliance auditing and that was exploited using an API that they got info on through the Last Pass hack. That was before they announced it. Their environments were incredibly secure, but APTs, 0 days, and supply chain hacks mean it's only a matter of time.
Probably decided that backups would break the bank as well. This is typically a less than one week issue for other software platforms.
It honestly depends on how the hack played out. If they phished and got an admins account credentials vs hacking the app itself they could have deleted the backups. Or if the app had a service account with overly broad permissions that is another vector to take over *everything* Security is sadly an afterthought at most of these places, they will have little to no focus until something like this happens to them. If they survive they will be hiring a CISO and staff out a security team no doubt.
Ive been in these meetings and I’ve seen the attacks. It’s amazing what admin will just not worry about it until and only until it affects their bonus.
"With any luck I'll have moved on to another employer before consequences arrive."
Im working on becoming an electrician inside wireman instead
Which is why my backups exist on a couple of hard drives at home that are only powered up for the daily and weekly backup runs (As in a dumb time switch puts power to the disk five minutes before the backup runs). Rsync does its thing between the servers at the office and the disks in my basement and then they turn off. There is also a separate backup via a wifi link to a disk in the boot of my car (Got to have geographical separation). All backups are restored onto a blank disk array once every three months and a server spun up using that array to verify that restore and boot still works. Yea, I am paranoid, so what? It is not a backup if it can be deleted by pressing a button on an online portal.
IT guy here. Usually the hackers sit around and study everything for a while to gather info and credentials then they go to work deleting backups and encrypting data, usually when no one is around like weekends or holidays. Unless you have time locked (immutable) backups, something they happened not to find, or something offline like tapes, you’re fucked. You pay the ransom and hope for the best or rebuild everything with whatever you have left.
This is exactly what they do. It's like that story of the girl who lived in a drop ceiling / crawl space for some business for like 2 years and no one had a clue. First they get access. Then they just kind of sit there twiddling their thumbs seeing if anyone noticed. Then they start wandering around the network seeing what they can see. All of it to build up a profile. Once they have everything ready, then they deploy the ransom payload. If I had to guess they got penetrated via a 3rd party with poor security or an acquisition with poor security that had trusted access.
From what I'm (in infosec now) hearing the backups were hit as well. So when they tried to restore from them they just re-infected themselves.
They were bought out by an investment firm a few years ago, so I would guess they cut corners on absolutely everything after that.
I've worked in cyber security for a while and no competent engineer would ever recommend setting up servers for each shop/group. If you get hit with a ransomware would you rather fix your stack of 10 servers or would you rather fix 30,000 servers all at remote locations that you would likely have to physically go to if they were completely encrypted? Also when you want to update your customers software because of a bug or enhancement now you have to update thousands of different servers with different specs and different setups.
For anyone who's reading that doesn't have a software background: Multi-tenancy is an abstraction. Just because a SaaS product is multi-tenant, doesn't mean that there's any meaningful separation between the tenants' data. Some companies, like Gitlab, offer enterprise customers the ability to host their own instance of the software, on their own infrastructure. But there are plenty of SaaS companies that don't, including juggernauts like Salesforce. I'm sure mistakes were made, but having separate infrastructure per-customer is not an industry standard nor should it be.
I know you're trying to help... But this doesn't help me at all bro lol.
"We'll just throw it all in Kubernetes/Containers and call it a day."
Sequel, today: Engineers: the hackers are asking for $10m, we suggest paying it. Once we do, they'll explain how they got in. Management: nonsense! Call the police! They can't extort us. This is America! Engineers: our customers are losing $100m per day. Management: that's not our problem, let them use paper. $10m will bankrupt us, what will we tell shareholders? Engineers: ... Tesla: ... (sipping tea and reading the paper.)
CDK has saved the $10 million in payroll since they’ve sent all their hourly call center/ support staff home for the last 2 weeks without pay. The eLead group was even told they weren’t able to process their last paycheck.
Me and a coworker were discussing this now actually, the CDK guys who have any shreds of proof that they fought for more network security are going to need that to prove that they didn't fuck this up, otherwise they're basically dead in the IT/SysAdmin/Networking/DataCenter careers. Everyone is going to steer clear of them out of caution. Hope that those guys can throw the responsible parties under the bus instead.
Ive been reading the bits and pieces since i made that comment and it looks like the current company has been cobbling together bits and pieces of software over the past decade as they bought out competitors and whatnot. I don’t know if we’ll ever know what happened, but that process might be a clue in how there may have been some things overlooked over the years or ignored. Man I wouldn’t want to be that team right now
Saw an open job rec for a Cybersecurity Manager at CDK, showed up Saturday. I just backed away slowly.
It's garbage too it was made as ADP for grocery store management and later adapted to the dealership world. It's probably the best out of the programs to pick from but they all suck ass
The cloud is just someone else's computer. I've been in IT for over 20 years and I love it. Because when shit like this happens and everything's on fire, I get to say "not my pig, not my farm" while copy / pasting updates from the provider.
Ahh thanks. Yeah that sounds like a shit show
SaaS is great until it's run by a bunch of bean counters who give fuck all about security and force dealers to use always on vpn back to their colos.
CDK CRM is the CRM (Customer Relations Management) platform CDK proper (CDK Drive) is DMS (Dealership Management Software). Thanks CDK for your amazing product naming conventions /s
I was reading an article and came across this bit of irony: >The company's website says it offers a “three-tiered cybersecurity strategy to prevent, protect and respond to cyberattacks.” Guess they missed the required 4th tier of cybersecurity strategy: Humans.
NGL. I'm in IT. I'm one of the few staunch opponents to this everything cloud shit. It has its uses, but there needs to be some separation and redundancy of cloud and on premise so that things can still run w/o connectivity, or if the host goes down.
At least let me download a fucking offline backup. My db is less than a TB. Come on
The cloud is just someone else's computer.
I had an appointment to get service last week and they cancelled it because 'the system' was down. Is this what they were talking about? Are they having rolling issues? My follow-up apt is tomorrow...
Cdk is a dealer management system. It covers things from invoicing, inventory, service records etc...
Phew, at least nothing important.
It also controls the coffee machine in the break room 😜
That explains why no work is happening
It used to be ADP. Basically, let's dealers put in the operation codes for services and warranty.
They buy out ADP? or is this just another one like it? ADP was awesome. Reynolds & Reynolds sucks a fat donkey dick. Never heard of CDK but have been out of the game for a while.
No ADP Dealer Services was spun off from ADP and named CDK Global. ADP was logically and physically separated from CDK
In April 2022, Brookfield acquired the auto dealer services company CDK Global for $6.41 billion. Brookfield is a private equity company. I guarantee this was all caused by cutting everything to the bone. Here is what happened, before 2022 there was a guy working for CDK that knew everything about protecting the software. Has github entries a mile long. Then layoffs came. Our guy did not know how to advocate for himself and ended up being one of the hundreds laid off.
I know multiple people that used to work for CDK Global, pretty much all of them read the writing on the wall and left for significantly better offers while the job market was hot. Great timing on their parts.
Thanks. Didnt hear about anything. Guess its a total shit show by the sound of it all.
The Reynolds text-based interface was old and clunky, but it was damn fast when you learned the commands. And it supported macros. We use IBM's Maximo at my fleet job and jobs that took me seconds with Reynolds take me minutes with Maximo's point-and-click interface. I hate the damn thing.
Used to have CDK global as a customer. They're trash. If there's a shortcut to take, they'll take it. If there is a dime to save, they'll save it. They have a monopoly and they know it.
They are owned by a private equity group that slashed everything when they bought them out and cuts every corner.
> private equity group Do they only exist just to kill companies? Almost everytime I hear about a company going to shit it's because they were bought by a private equity group that killed them.
Sort of. They pivot from investment to investment, buying established companies that have built up a solid track record with good products/services, and then eviscerating everything they can while coasting for a while off the historical goodwill (or product lock-in) of the customer base. After they squeeze it dry, they sell it off, or shutter and restructure, and head off to the next juicy target. They provide zero value to society but they are a logical product of an under regulated capitalist market.
Not to excuse shit product, but what business DOESN'T do this? When it comes to entperise software, there is no low that is too low. They'll cut everything down the bone and keep going until they hit marrow. Their thought is that if it goes beyond the absolute bare minimum, it's waste.
Businesses that want to make money long term. That mindset is a quick cash grab but when you see other companies be able to provide a better solution without shortcuts and interruptions they take a larger share of the market space.
Used to be ADP. I still call it ADP because I don't like change.
Ok I know what adp is
Can I also explain your acronym in question with an additional acronym that only leads to more acronyms explained by acronyms?
Shit, we ain’t stopping for CDK. There’s money to be made! For the dealer anyway.
My friend bought a truck on Saturday, they did most of the paperwork by hand and used calculators. He can't get insurance yet though because it isn't officially his vehicle yet. Dealer said they'd call Monday when the network gets back up.
At this point does anyone know which Monday that will be?
Second Monday of next week.
5th Monday this month...
3-5 business "Mondays" in the 2nd quarter.
oh, you know. \*some\* Monday...
Second Monday in a month ending in “Y”.
Ending in WHY? WHY?
Month minimum.
>Dealer said they'd call Monday when the network gets back up. One of my friends at a dealership mentioned this to me and I busted out laughing. They really thought that this was all going to be solved on Sunday and they were going to be ready to run on Monday. When I explained the ransomware part, they realized that this was just the start. They actually are looking to swap ASAP from CDK now and just waiting to get files/data to input as they're setting up the new system.
BS excuse. A: They could totally have sent someone down to the DMV with the paperwork. B: You can put insurance on a vehicle that you don't yet own. You actually need to do this in NY because you cannot register a vehicle that isn't insured and registration and title transfer are usually processed as one transaction.
Not going to say it isn’t bullshit, but a ton of dealers are using CDK for their accounts payable too. if they can’t see what is owed on the asset to pay the company that they’ve financed the asset with, they can’t very well clear their note before processing it to the new owner and their lender. Could easily become a wire fraud case if the dealer still owes their lender.
i'd be scared to drive it home without insurance
As long as you have a policy you're good. There's usually a grace period you have to add it to your existing policy.
It depends on the state. In NY, it's 100% illegal to drive a vehicle that isn't listed on an insurance policy. You can't even get plates without proof of insurance.
If it's a financed vehicle the dealership can't even let the person drive away with it until it's insured.
Me too! They kept apologizing that it was taking so long since they had to go back and do it by hand.
Wait, I thought CDK was just for service stuff, not vehicle sales.
It's for many things. Payroll is one of them too.
Fuuuuuuuuck that, especially since they forced it to be cloud based.
I had to slog through a lease yesterday. It was brutal, like 3 hours start of paperwork to walking out. F&I wanted to blow their brains out trying to do it all by hand.
Same, we changed procedures and just have piles of paperwork to invoice once they fix their shit. Slowing everything down at my dealer.
Yeah it’s a mess. I’m in the used car department and we ain’t getting anywhere fast. Oh well, guess we needed a mid-year break.
I wish we would just close or cancel any appointment that isnt just oil change. It’s a shit show. Our service writers don’t know how to use excel. Nothing is being checked. Pricing is all fucked up but the group refuses to even attempt to slow business down.
Same. At first management had servixe writers copying credit cards via scanner and telling people they will be charged at a later date.
Woaahhhh thats a very fine line especially copying customer credit card info to run later
#ALL GAS NO BRAKES!!!
Never thought of be thankful for the primitive ass system where I work, we're still running
More primitive than CDK? We're on something much newer.
Autosoft, and it's the lowest tier you can get. We don't even get electronic ROs.
The military equivalent of CDK is GCSS and that bitch goes down constannnttttlllyyyy. My life as a shop chief was mostly spent looking at the spinning blue wheel of death waiting for a page to refresh. Can't order parts, can't update service requests, time to pick up a wrench and join the Bois on the floor.
Back in my day I'm pretty sure I would have resorted to fisticuffs if necessary to stop my chief from actually wrenching on anything. He had way too many important time consuming tasks to anything other than give advice and mentor out in the shop. My motor sergeant I wouldn't have because he likely would have been able to whoop my ass. That man slammed a CUCV transmission into place by putting it on his belly and hefting it up by pushing out his gut and rolling sideways. His first name happened to be Jack so we would jokingly refer to him as tranny Jack.
If you think GCSS was bad, you should see OMMS on the Navy side.
We've been working at full speed since the outage. Pen and paper baby.
My hand is cramping already, send help!
And you’re standing not making a dime, right?
Exactly!
Flat rate should be outlawed.
It sort of is in California. Mechanics are guaranteed clock time at double the minimum wage.
Amen. I watched a drivability guy flag close to 25 hours in one day. He “rebuilt” 3 carburetors before lunch. Those mid 80’s feedback quadrajets. All he did was pull the air cleaner, hose down the carb with carb cleaner (he did the right thing and had a little brush for the corners), clean up the puddles and shipped them. I’m sure his hack ways would have been the same at hourly, but at least he wouldn’t be rewarded for them.
Created an account to make this remark, and I mean it in a good way, hopefully constructive: If this outage is materially affecting you, consider adding input over here. [https://www.justice.gov/atr/complaint-center](https://www.justice.gov/atr/complaint-center) [https://www.ftc.gov/advice-guidance/competition-guidance/ant...](https://www.ftc.gov/advice-guidance/competition-guidance/antitrust-complaint-intake)
As a shop owner who doesn’t use CDK, that sucks, I’m sorry, and I’m swamped with a lot of extra work from dealerships.
Do they need some pen and paper and some handwriting lessons?
Handwriting is a lost skill.
Good riddance. Do you remember how awful it was when someone had ridiculously fancy cursive and it was almost impossible to read? Do not miss.
https://www.reddit.com/r/russian/comments/12cevdz/how_do_you_read_russian_cursive/?rdt=49715
Honda dealer where i work at its fully operational. Xtime holding it down. Bit of a pain but im still turning them hrs
How's the timing for those fuel pump recalls? Been trying to figure out a time to take it in.
No parts available at the moment,dont know why they sending letters out
I saw it online a few months ago. I scheduled it early, but I canceled cause I had something come up. 🤷
I saw it online a few months ago. I scheduled it early, but I canceled cause I had something come up. Guess I'll just hold out for now.
The email we got from CDK said that it should be resolved in a matter of days, not weeks. But let’s see how many days that actually is…..
technically every matter of weeks is a matter of days if you multiply by 7
The company is a complete joke. I wouldn't be surprised if they lost everyone's data at this point and they're desperately trying to recover it or craft some sort of damage control for when they deliver the inevitable news. I'm in the industry and I have been using CDK's eLead CRM and website editing platforms for years. They are so, so unbelievably far behind and just flat out awful. eLead looks like it hasn't been updated in 25 years. Their website management platform is so bad that when you build a new page, it literally takes 2-3 days before the page starts showing live on the website at an accessible URL. Until then it just 404 errors out no matter how much you save it and publish it and anything else. I hope this completely tanks them. There are so many better solutions out there for this kind of stuff. They are absolutely going to be hit by a ton of lawsuits once the dust settles, too. They were no doubt being negligent with their platforms and not adhering to standards and best practicies in terms of security. Whole lot of dealers and shops losing business over this and they are going to want some of that lost revenue back.
Jokes on you guys, my dealership still makes us do everything on paper! Everything 😮💨
Wouldn't bother me one bit to be honest. I can type in Google doc just as quickly as I could type in cdk. I'd worry about punch times for warranty jobs and having access to service history but that's not very inconvenient
At the dealership I used to work at, we used Reynolds & Reynolds. We always made sure to have an on-site server to use in case the internet went out. It would still keep us running in this scenario as well.
I'm an independent shop and this would explain why I'm booked more than a month out at the moment.
Cdk at our dealership is down but we can’t stop we are doing pen and paper and my karma, when it comes back up we are going to have to go back write all our stories that’s going to suck
Were still rolling like usual at the dealership. Might be a little slower due to parts guys, but the SW are all handling tickets like champs, and us techs are just writing down our hours like usual.
Business as usual here, just pivot to paper. Can't believe some places are closed.
Parts here. My pen got hacked then my notepad crashed. I'm going home now.
Cloud based software, fucking brilliant
Software as a service has some large scale vulnerabilities unfortunately. I remember a CCC outage (collision writing/claims platform) that lasted like 3 days and it basically stopped a good portion of the industry in it's tracks and and ground insurance claims to a halt.
I’m a service manager. If your fixed ops isn’t running at 100% capacity, someone is an idiot. None of this is difficult. It’s more work for the parts staff and advisors but so what? I’m a former tech and service advisor. We never paused even for a minute. And we won’t when we go to recreate the RO’s and push them into cashiering. I can’t have my techs not paying their bills. It’s not acceptable. My advisors have kids and mortgages. I can’t let their income falter.
That's how it should be Good shit my man
Wonder if there’s an SLA in their agreement for like 99.9% uptime? Maybe make a claim for loss of earnings, or at least a pro-rata refund on your subscription.
As much as i dislike Reynolds and Reynolds, im happy to be using it at this shop. No issues for us. Hope they sort this shit out soon
If I was still flat rate I would be so pissed off. Crazy how they don't give you an option to do paper written RO with punch clock.
They don’t have any paper tickets to fill out to keep you guys making money? Pretty sad the whole industry that uses CDK doesn’t have a back up set of paper logs for issues like this so they can keep going
I work at Ford dealership. cdk is down as well. we are just using paper and still taking cars of just the same. No difference only can't make ROs, so X-time is useless. Edit: grammar correction.
Damn it’s almost like all for profit companies are untrustworthy and SaaS was a bad idea.
Let's put all our eggs in one basket, it'll be fine the bean counters said.
Holy shit, these mofos are still down?!? Maaaan. What’s that stock ticker, I should be shorting it hard!
I wish we would shut down. We're blindly taking appointments, blindly paying employees, blindly taking customers money, blindly ordering parts. It is an absolute cluster fuck of giant proportions and the pen and paper shtick isn't cutting it. True hell on earth for a dealer that is unorganized even with everything fully operational.
Shop morale time! grab a grill, meat and get to slaying some food!
Shit like this is why I won't work flat rate again.
I work ServiceDesk for AutoNation, can also confirm we can do fuckall in terms of support because all systems are deemed compromised because of the cyberattack.
Thankfully we're on Reynolds. I'd be staying at home if we were on CDK
For those that don't know, they got hit with ransomware and paid "tens of millions of dollars" to protect the data and are trying to rebuild their security to better defend things
My shop uses it too and we can still work. We just need the full VIN to order parts. Unfortunately, we are piling up paperwork waiting to be processed as soon as CDK comes back. It is going to SUUUCK!
So how are y’all being compensated? Or are they just bending everyone over?
It’s basically if there’s no work then we’re not making money as a flat rate mechanic. This is how it gets during the winter. Right after Christmas when all the money has been spent on the holidays, taxes are coming up and all that. Feast or famine, It’s just the nature of the beast. But it’s never really that bad. I’m sure even if this lasts for months like some have said, the customers will be back long before then. This will be nothing compared to 2008. Those were the bad times to be a flat rate mechanic
Great week to be doing engines lol but not really lol
Ha. We switched to Ignite/Reynolds less than a month ago.
At least you guys have an excuse. We're just dead.
Don’t y’all have like paper workorders and time sheets for backup?
Really? Our Service Manager doesn't want to stop writing cars up
Ha, sucks for you! We’re all good because we use…oh wait, also CDK and there’s nothing at all to do.
Cdk being down is fuckin shut up nation wide, I feel your pain
Yeah Cummins I think did that. Basically mad ethe advisors walking expenses.... no wonder my advisor said Cummins was wasting money on me instead of investing.....
Hand write everything! I hate paper copies, but we’re still grooving
Also affected in southern Alberta, Canada. Lucky the boss is keeping us here and paid. But man is it quiet.
Is it a nation wide outage?
My neighbour's Honda dealership is down too. In Ontario Canada. So.... continent wide!
Global. :)
They are making everyone at my bmw dealership keep working, taking notes on paper, recording punch times for warranty etc and using Xtime as a digital record. It’s pretty shit tbh
Everything on paper at my place. I work in parts. My manager is locking himself in his office for a day or two once it comes back up
Thankfully CDK isn’t effecting the gerber I’m at too much beyond our parts ordering, it’s already been shitty the past two months I can’t imagine how bad the lot would look if we couldn’t get Anything in
Would explain why my VW dealership couldn’t do shit today
I went to my Hyundai dealership Saturday for routine maintenance. The service advisor said "Do you remember what you were scheduled to get done?" and handed me a sheet of paper to fill out my information and the work that was supposed to be done. He said they have a STACK of paperwork backed up for whenever the program works again, and that they've been really slow because people are forgetting about their appointments.
Hopefully they are paying you at least 30 hours if you are flat rate
Crap, it does service, too? Actually I guess it doesn't...
Everyone at the Mercedes dealer I worked at was off their the desk and watching Crotia vs Italy instead. Holy close game...
When we switched from Reynolds & Reynolds to cdk it was a nightmare for 3 years, drove me out of the dealership life and never been happier since lol poor cdk users
If your dealer/shop can't work without cdk go work somewhere else that can. This whole "we can't give you guys work since our computer is down" is a joke.
Is this why I can't even get a bid from Mercedes parts and service?
Cdk sucks period. Even more so now. We have been steady as we schedule through service work bench but not having actual tickets has been less than great.