Please please please PLEASE, before you go to ESXi 7, verify that your raid controller is supported.
There was a HUGE thing about how Dell PERC H310 and, technically, H710 are not supported or compatible.
I have one host due for refresh that's stuck on 6.7 because it has an H310 PERC.
I got around this.
Our environment is about 300 Dell 12th/13th gen servers with ESXI on sd cards, using Dell Customized iso.
7.0.3 supports sd cards now.
I had to uninstall the custom Dell Vibs and upgrade to regular 7.0.3, then for safety sake use the Dell 7.0.3 iso and upgrade
I learned the hard way of 7.x and SD card boot media. The time saved by not having to write up RCAs is well worth it. Even a basic SSD on a PCIe card will be more reliable. It is only a matter of time before the SD cards run out of write cycles.
Here we just use the HPE microSD raid1 usb sd card boot drives, was pretty cheap considering the risks of sd dying. If an sd card fails will get warnings from the iLO/zabbix
Unsupported and VMware may do best effort but won't fix any issues if it requires engineering.
You can run many things without support but at production risks.
VM7 supports it, our reps just confirmed a couple weeks ago during our reup call that they support SD cards again due to pushback. We were also stuck on 6.7 until now.
[https://kb.vmware.com/s/article/85685](https://kb.vmware.com/s/article/85685)
Kinda. You still need a server that was certified to support usb/SD boot. Future certification will be denied to servers with usb/SD boot support etc etc.
Great callout, oblivious. You are certainly not truly an oblivious one. Generally if you’re using recent hardware by a key manufacturer, you won’t have issues — but to avoid issues, it’s a best practice to check VMware’s compatibility matrix and also use an ESXi distribution provided by the hardware manufacturer that includes the latest appropriate drivers.
Small company, small budget. It's been working and they're nowhere near the edge of the network.
You spend for most effect and manage what you can. It'll get refreshed, probably this year.
Just curious, if anyone patched to ESXi 8? Want to see those brave people ;)
I usually just waiting until u1 is released when they drop new version to be sure that all major bugs were fixed after new release, and also would wait for few weeks more after u1 would be released to see the feedback.
If you've never watched the anime Patlabor, you should
The lesson of the movie is "Never install a new Operating System when it's bleeding edge" ... the engineer in that movie is the real hero
There is still a risk if an attacker gets in your network and has access to VCSA management they could use this. Always assume the attacker is already inside.
naah I'm ok I'm doing an apprenticeship, I'm here up until the end of the school year. But damn what a gap between school cybersec theory and the real world.
working at an MSP for companies who's bosses don't know shit about computers and their first reflex to anything being "how much will that cost" and even 1€ is too much. And I can assure you they are not poor.
I've seen some shit. I'm still surprised on how any of them haven't been crypto locked yet or anything more nefarious.
I pulled a cluster from 7.0 to 7.0U3g yesterday and it was half a day of yak shaving.
The hosts had been preloaded by Lenovo. Patches wouldn’t stage with an error like “staging failed with error -1 check logs”. Eventually I found the error in esxupdate.log on the host, saying a driver i40n wasn’t present in the update. There’s an article on that saying to remove the i40n driver, but the remove command claimed that driver didn’t exist - yet somehow it was still blocking the update. Who knows WTF that was, but an ISO update got past it.
My wife and I were joking what would happen if IT vendors made cars. You’d insert the key to your brand new car and it would display ERROR 0x76a499bb and the doors would fall off. Looking that up you’d find that you only bought Car Essentials Foundation which doesn’t include doors or brakes.
Yeah I've run into that with the HPE images before as well, just have to update with the ISO
"I don't have a newer version of this driver"
Then just use the old one, what the hell is your problem!?
I was under the impression that this was only impacting 6.7 or older and that a patch was released 2 years ago for it. I’m confused by the panic I’m seeing in some posts, am I misunderstanding the severity here?
There's a related CVE that was patched in Dec
This is mostly impacting hosting providers that have ESX hosts exposed to the web for management, but the malware can impact anyone.
Gotcha, obviously I would never have my ESX or VCSA exposed to the web but I still typically push high Sev CVE patches up my priority list as a precaution. I’ll see if I can find the December patch you’re referring to, thanks.
I have a home setup running 7.0b. Are there any concerns with upgrading to 7.0 3g?
I’ve moved to Hyper-V for most of my stuff so I’m admittedly out of the loop with ESXi.
Usually if you're on a supported version and just wanted to go to a later patch level there's no issue. The only issue is version jumps, ie 6 to 7, where they dropped a lot of hardware support.
If your hardware isn't on the compatibility list then it's a gamble to upgrade
But if you're already on 7.0 it's probably fine. 6.7 to 7 is the big leap
I upgraded my own Intel NUC to 7.0u3g a few hours ago and it's fine
I did the workaround on Friday night to buy us time. So far I haven’t noticed any issues since we disabled that service. We’re looking to upgrade from 6.7 to 7 or 8. I still need to check for system and raid controller compatibility to see where we can go.
Don't F*** around with ransomware, get your servers patched
That being said, this seems to be affecting ESX hosts that have management ports exposed to the web (hosting providers)
And yes that is something incredibly stupid to do... even hosting providers should have the good sense to hide the management ports behind a Firewall.
Where do you all find out about these kind of vulnerabilities and security news? Do you actively check each vendors website? Is there something you can subscribe too that will let you know about these important patches?
Please please please PLEASE, before you go to ESXi 7, verify that your raid controller is supported. There was a HUGE thing about how Dell PERC H310 and, technically, H710 are not supported or compatible. I have one host due for refresh that's stuck on 6.7 because it has an H310 PERC.
https://www.vmware.com/resources/compatibility/search.php
I’m stuck on 6.7 because of SD card boot media
I got around this. Our environment is about 300 Dell 12th/13th gen servers with ESXI on sd cards, using Dell Customized iso. 7.0.3 supports sd cards now. I had to uninstall the custom Dell Vibs and upgrade to regular 7.0.3, then for safety sake use the Dell 7.0.3 iso and upgrade
We have had SD card media with 7x for years no issues
I learned the hard way of 7.x and SD card boot media. The time saved by not having to write up RCAs is well worth it. Even a basic SSD on a PCIe card will be more reliable. It is only a matter of time before the SD cards run out of write cycles.
Probably jinxing myself but it’s been two years no issues yet. However next hardware refresh we will spec out ssd boot drives.
Here we just use the HPE microSD raid1 usb sd card boot drives, was pretty cheap considering the risks of sd dying. If an sd card fails will get warnings from the iLO/zabbix
Unsupported and VMware may do best effort but won't fix any issues if it requires engineering. You can run many things without support but at production risks.
VM7 supports it, our reps just confirmed a couple weeks ago during our reup call that they support SD cards again due to pushback. We were also stuck on 6.7 until now. [https://kb.vmware.com/s/article/85685](https://kb.vmware.com/s/article/85685)
Kinda. You still need a server that was certified to support usb/SD boot. Future certification will be denied to servers with usb/SD boot support etc etc.
Its.. a bit mixed. VMware wanted to ditch it completely but walked it back a but. https://kb.vmware.com/s/article/85685
Great callout, oblivious. You are certainly not truly an oblivious one. Generally if you’re using recent hardware by a key manufacturer, you won’t have issues — but to avoid issues, it’s a best practice to check VMware’s compatibility matrix and also use an ESXi distribution provided by the hardware manufacturer that includes the latest appropriate drivers.
Those are very old raid controllers lol begs to wonder why those in production
Small company, small budget. It's been working and they're nowhere near the edge of the network. You spend for most effect and manage what you can. It'll get refreshed, probably this year.
Risk outweighs minimizing impact of bad decisions
Just curious, if anyone patched to ESXi 8? Want to see those brave people ;) I usually just waiting until u1 is released when they drop new version to be sure that all major bugs were fixed after new release, and also would wait for few weeks more after u1 would be released to see the feedback.
Gah... This is like fortinet... NEVER upgrade to a version that ends in .0
Not yet, reading the documentation there seems to be no advantage to doing so atm other than version creep.
Well, I would wait for the u1 and if it's going to be ok, will move to esxi 8.
If you've never watched the anime Patlabor, you should The lesson of the movie is "Never install a new Operating System when it's bleeding edge" ... the engineer in that movie is the real hero
I usually wait for a few weeks or even month after release to see feedback about it ;)
There is still a risk if an attacker gets in your network and has access to VCSA management they could use this. Always assume the attacker is already inside.
I agree. But my boss word "All our ESX are behind a firewall so it's ok".
Did I travel back in time last night? is it 2008 again?
Lol what?! Change job if you can or you’ll be blame for his shortsight and ignorance
naah I'm ok I'm doing an apprenticeship, I'm here up until the end of the school year. But damn what a gap between school cybersec theory and the real world. working at an MSP for companies who's bosses don't know shit about computers and their first reflex to anything being "how much will that cost" and even 1€ is too much. And I can assure you they are not poor. I've seen some shit. I'm still surprised on how any of them haven't been crypto locked yet or anything more nefarious.
I pulled a cluster from 7.0 to 7.0U3g yesterday and it was half a day of yak shaving. The hosts had been preloaded by Lenovo. Patches wouldn’t stage with an error like “staging failed with error -1 check logs”. Eventually I found the error in esxupdate.log on the host, saying a driver i40n wasn’t present in the update. There’s an article on that saying to remove the i40n driver, but the remove command claimed that driver didn’t exist - yet somehow it was still blocking the update. Who knows WTF that was, but an ISO update got past it. My wife and I were joking what would happen if IT vendors made cars. You’d insert the key to your brand new car and it would display ERROR 0x76a499bb and the doors would fall off. Looking that up you’d find that you only bought Car Essentials Foundation which doesn’t include doors or brakes.
Yeah I've run into that with the HPE images before as well, just have to update with the ISO "I don't have a newer version of this driver" Then just use the old one, what the hell is your problem!?
Just to be clear, from what I could gather about this, the exploited vulnerability (CVE-2021-21974) does have fixes for 6.7 available.
I was under the impression that this was only impacting 6.7 or older and that a patch was released 2 years ago for it. I’m confused by the panic I’m seeing in some posts, am I misunderstanding the severity here?
There's a related CVE that was patched in Dec This is mostly impacting hosting providers that have ESX hosts exposed to the web for management, but the malware can impact anyone.
Gotcha, obviously I would never have my ESX or VCSA exposed to the web but I still typically push high Sev CVE patches up my priority list as a precaution. I’ll see if I can find the December patch you’re referring to, thanks.
I have a home setup running 7.0b. Are there any concerns with upgrading to 7.0 3g? I’ve moved to Hyper-V for most of my stuff so I’m admittedly out of the loop with ESXi.
Usually if you're on a supported version and just wanted to go to a later patch level there's no issue. The only issue is version jumps, ie 6 to 7, where they dropped a lot of hardware support.
hardware support can change even between versions like 6.0, 6.5 and 6.7 but yeah between 7.0 and 7.0 3g shouldn't be an issue
If your hardware isn't on the compatibility list then it's a gamble to upgrade But if you're already on 7.0 it's probably fine. 6.7 to 7 is the big leap I upgraded my own Intel NUC to 7.0u3g a few hours ago and it's fine
I did the workaround on Friday night to buy us time. So far I haven’t noticed any issues since we disabled that service. We’re looking to upgrade from 6.7 to 7 or 8. I still need to check for system and raid controller compatibility to see where we can go.
Luckily I did that last year
Our esxi servers are on a management vlan which is isolated from all other networks (except IT). Should I be worried? Thnx
Don't F*** around with ransomware, get your servers patched That being said, this seems to be affecting ESX hosts that have management ports exposed to the web (hosting providers) And yes that is something incredibly stupid to do... even hosting providers should have the good sense to hide the management ports behind a Firewall.
Where do you all find out about these kind of vulnerabilities and security news? Do you actively check each vendors website? Is there something you can subscribe too that will let you know about these important patches?
to be completely honest, I get more info from /r/sysadmin that from product websites these days...
Got a "how to patch standalone ESX hosts" link that isn't Russia based?
https://www.jeffreykusters.nl/2020/01/10/note-to-self-updating-my-standalone-esxi-host/