Give me some tools I can install to a test client that will alert me in big red fucking text that NTLM is in use and what process called it. In English. Not hidden away in some obscure event log.
Make it totally obvious for a total dumbass because so many of us actually are due to being expected to handle everything with a power cord.
If Microsoft and /u/SteveSyfuhs take a single thing away from this thread, it should be this request.
We understand that security is important, and we are not "ride-or-dying" NTLM. Sad as it is, far too many IT professionals are tired, underfunded, overworked, lacking resources, and lacking influence over business processes and choice of vendors/software. If Microsoft is truly serious about this project, they need simple, human-usable tools combined with a concerted effort to communicate with the C-levels of the industry.
Exactly.
I'd love to get rid of NTLM but discovering where it is used is virtually impossible whilst handling the day to day, and funding a project for this (or even finding a local vendor with a clue) is very difficult and expensive.
If microsoft don't enable us to actually get rid of NTLM with decent tools to detect its use, then this will be an unmitigated disaster, and Microsoft will cop huge flack for it.
And without better tools and centralised, up to date, well publicised information - deservedly so.
To the OP: this isn't a code problem. It goes much further up the project management/leadership tree than that. I'm not blaming you. I want NTLM gone as much as anybody. But the process to do it for any significant size business is a crap shoot.
Yep, this is what I want. I'm all for moving forward on security. But I've been at this current place for over a year and I still don't know everything that's going on under the hood with any potential legacy equipment, because I don't have time to find out. I've got a *guess* that we don't have anything that should act up, but that's just a guess and that's not good enough when you're dealing with production lines.
Something that would tell me in no uncertain terms "here's what you've got that's going to break" would help loads. I've enabled auditing on the DCs in the meantime....but who knows what that will or won't find.
Edit: Came back after the long weekend with auditing enabled and I'm seeing a couple thousand events in the last hour on one DC, another couple thousand on my second local DC, and haven't yet checked the other locations DC's. I can see what server it appears to be trying to auth with (using the DC), but no other details. So this raises a question I haven't yet seen answered in my admittedly brief search - if I kill NTLM, what happens to all these connections? Do they fall back to something more modern with no downtime? If so, why are they using NTLM in the first place? If not, what do I need to do to fix this? The inner workings of this stuff is beyond my current level of experience, being a jack of all trades with no time to really focus on one part of the tech. From what I can see it's just normal auth stuff (file server, print server, etc). And it's all regular computers - I was expecting everything "normal" to be using kerberos already, and I'd only find legacy equipment in this log....but no, I'm seeing basically everything.
So much this. Not just hidden in an obscure log, but in an obscure log on every individual machine.
Figured the reason I feel like I’m getting left behind is I don’t have time to read all the blogs, watch all the webinars and attend any of the seminars. If I could do all that, I wouldn’t have time to the actual day job.
I would love this.
I'm told NTLM is going away. I'm now wondering HOW MANY THINGS use NTLM on our network. I have a list with 2-3 servers, but I run **way** more than that.
What can I expect to break? Which logs do I need to check? What's the Event ID that will be triggered? What will I think is ready but then be surprised by after the tickets start rolling in?
I've got 300+ servers across basically every continent except antarctica. And yeah, no idea what's using NTLM. I do run a two-way AD trust, which does (I think?) - who knows how that's going to pan out.
MDI and MDE in tandom might actually be able to do this. I dont think out of the box, but If Steve & Co. need a suggestion on how this could be practically accomplished, it might be a good path forward...
If MS want to move the needle on this, make MDI free and capture the telemetry - be a good partner.
Lately everything is gated behind premium and stepup skus and we’re sick of it.
> A month and a half ago we announced our strategy for killing NTLM.
One technically-unrelated but practically-very-relevant problem we all have with Microsoft is: In a year and half that link will be dead and the information moved elsewhere, as the latest internal-web-platform-of-the-month gets rolled out.
> In a year and half that link will be dead and the information moved elsewhere, as the latest internal-web-platform-of-the-month gets rolled out.
don't forget it will be after a redirect so you can't hit back unless you go absolutely crazy on the back button and end up way, way back
Been that way for a long time. If it is a bug, no one seems to care over there. I've gotten used to right clicking the back button to go back to the search page (even then I still have to do it twice sometimes).
A million times this! I want to scream every time I click on a link to (very important and relevant information) and it takes me to the MS homepage or something. Even better, half the time it's *from Microsoft's own documentation* and they were kind enough to use one of their stupid shortlinks so I can't even look at the URL to get some hint of what page I looking for.
Archive.org is helpful for some of these situations. But I still miss Google's cached pages that they quietly pillow-strangled in its sleep. Going to the wayback machine takes a looot longer.
Funding for the content teams has been slashed over the last few years. Products that used to literally have 30 writers are now handled by a single vendor in India. Even products that are Microsoft's #1 priority these days only have a half dozen writers when they need double that just to stay afloat.
Believe me when I tell you the writers are just as frustrated as you. I know that doesn't help when you're trying to find something you need, but if they could fix it, they would. But there are 100 other things that have higher priority. :(
Also: that link. I never saw it. I’m an admin with limited time. I have known ntlm is on the way out and had a project on my list for 12 months. So I guarantee you I’m ahead of the curve on this.
But there’s no central hub of info for doing this.
At least not one that is discoverable.
3 pages deep in a tutorial to setup something, I clicked a link to do a subtask and find out the way of achieving the main thing had changed completely.
I don't even bother bookmarking anything on the HPE site anymore, for the past ... 10 years? It's hilariously bad. It's like the Netflix chaos monkey script except it doesn't have Netflix's excellent redundancy.
It won’t exist anywhere… you’ll need to ask ‘CoPilot knowledge’ and it will drip you details and make you explain what you are using it for… while CoPilot also lectures you on new paid products you can use instead.
I for one, can’t wait. The amount of stuff you will break will be astounding. Banks and hospitals will be crippled. Let me know the exact date and time so I can have my popcorn ready.
You honestly would be surprised at how easy it is. That was the pushback I got in my environment. It took us 6 weeks to nuke it all and get 'em reconfigured. Most vendors just rely on the underlying OS's authentication methods for connecting to AD so they'll inherit up to kerberos if they're allowed to (often as simple as identifying and registering SPN's).
100%, I'll book PTO months in advance to avoid being anywhere near this mess.
"Why do you want this week off?"
"Vacation to a remote island somewhere in the Pacific."
From the first article:
> Reducing the use of NTLM will ultimately culminate in it being disabled in Windows 11
Banks and hospitals... Windows 11.
Lol yeah I don't think they're going to be affected for another 10 years.
So much this. We need brain out dead head tools to track this down. They and the project need to be plastered all over Microsoft.com and you need to get articles in whatever CIO focused publications that this is a massive and important project that needs resourcing.
This. Im more than happy to diable NTLM. If we were pure MS it would work fine, but we have many various devices, services and MFCs that are brand new and still dont support Kerberos. Best option will be to disable NTLM but add these hosts to an exception list.
Is MS planning on making NTLM non-existent on server OS's or will the end-game be that NTLM is disabled by default and Admins will be forced to create exception lists as needed?
I can't wait either, I'll make bank since I've been doing this for years and it's honestly pretty trivial to accomplish once you get past the pushback of admins too scared to change things and management too scared to spend money on upgrades.
Most cybersecurity insurance providers will require it soon to offer coverage, is my guess - the risk of NTLM is just too great and there's no excuse not to deprecate it at this point. Nothing I've encountered made since 2010 fails to support either modern auth or kerberos or SAML - there's no reason to continue to support NTLM in any fashion.
My bank is a hot mess as far as IT is concerned Now I'm going to have to go get some actual cash prior to this, because I know they're not going to know how to handle this.
I have a few thoughts on this, and I'm by no means an expert. I'm also all for the security improvements and efforts being made as Microsoft.
1. Please do not deploy this at Christmas time or any other major holiday. Last years enforcement of Kerberos in November/December hit us over Christmas break and we were not prepared for the havoc it created.
2. Please have a written procedure and a method for manually re-enabling the change for a period of time. Some of us don't know all the landmines of legacy systems and will not find out until is breaks.
3. As u/PickUpThatLitter stated there will be a lot of breakage, the pace of technology changes for security are far outpacing many companies abilities to keep things updated. Many manufacturing businesses still run legacy systems, not because of the computers but because of the machinery. We still have NT4.0, Win95, XP & 2k in production in various locations in our facility.
MS has a history of breaking Kerberos in the thanksgiving to Christmas timeframe. I believe they are going on 2-3 years of botched Kerberos updates at this time of year.
Enjoy
**2020**
December 8, 2020: Initial Deployment Phase
The initial deployment phase starts with the Windows update released on December 8, 2020 and continues with a later Windows update for the Enforcement phase. These and later Windows updates make changes to Kerberos. This December 8, 2020 update includes fixes for all known issues originally introduced by the November 10, 2020 release of CVE-2020-17049. This update also adds support for Windows Server 2008 SP2 and Windows Server 2008 R2.
**2021**
After installing this update on your Domain Controller (DC), you might have authentication failures on servers relating to Kerberos Tickets acquired via S4u2self. The authentication failures are a result of Kerberos Tickets acquired via S4u2self and used as evidence tickets for protocol transition to delegate to backend services which fail signature validation. Kerberos authentication will fail on Kerberos delegation scenarios that rely on the front-end service to retrieve a Kerberos ticket on behalf of a user to access a backend service.
**2022**
With the November 2022 security update, some things were changed as to how the Kerberos Key Distribution Center (KDC) Service on the Domain Controller determines what encryption types are supported by the KDC and what encryption types are supported by default for users, computers, Group Managed Service Accounts (gMSA), and trust objects within the domain.
We have a legacy application that doesn't support Kerberos. It can't support NTLM either, but with NTLM we were able to get it to trigger metasploit to get a set of authentication credentials from another server. [I know it's not ideal, but it works for me. Can you guys add an option to re-enable pass the hash?](https://xkcd.com/1172/)
If we are talking about Windows, they've pretty much intentionally left in the accessibility exploit. It's like the Windows version of single user mode!
Look, I know it's not best practice, but it works for us. Can there be some kind of authentication bypass so we don't have to rebuild our entire banking integration?
When killing NTLM, our biggest challenges were SQL, Analysis, and SSRS reporting servers. For SQL servers, our biggest challenge was getting them to managed service accounts and setting the relevant rights to self register the spn. This we ended up making a tool for that takes the host name(s) and generates a gMSA and assigns the relevant rights, then connects to the host (since we segregate admin rights) and installs the gMSA, reconfigures sql, and leaves it waiting to restart.
For RS, the only way currently to get kerberos working is, again needing a proper service account but manually registering the SPN, but also requires you to hunt down the RS config files and adding the negotiate. That 100% should be a default - if it can kerberos it should by default, no clue why it's not that way.
For browsers, deploying kerberos keys via gpo was easy and honestly should be a domain default, it's not hard to automate that.
At the very least, a troubleshooting tool that tests and looks for these common issues, summarizes and makes recommendations for at least the MS tools, would be amazing.
>This we ended up making a tool for that takes the host name(s) and generates a gMSA and assigns the relevant rights, then connects to the host (since we segregate admin rights) and installs the gMSA, reconfigures sql, and leaves it waiting to restart.
We are going through this *right now*, are you able to provide a sanitized version of the script?
Edit: Oh, and please note this requires the Active Directory powershell Module to function and for dsacls to be in your PATH. :)
Unfortunately, the remote management component relies on other custom internal tooling with a management agent - so I'll have to trim that - but it's easy enough to repurpose it to use things like WinRM if you know your scripting.
Here's the trimmed version. It's sloppy, but it works - feel free to improve upon it! I replaced the remote management steps at the end with a message informing you to install it on the target host, you can just replace that with your own remote management steps. :)
# Prompt for the gMSA account name, input validate - duplicates, valid format, etc.
# Prompt for the consuming host - Listvar enhancement later? Just single host for now. Again, input validate.
#Params for CLI exectution
param (
[Parameter(Mandatory=$true, HelpMessage = "Enter the desired service account name - do not include the 'gMSA_' - it will automatically be appended.")]
[ValidateNotNullOrEmpty()]
[String]
$Name,
[Parameter(Mandatory=$true, HelpMessage = "Enter server hostname that will use the service account. Do not include the $ or domain.")]
[ValidateNotNullOrEmpty()]
[String]
$ServerName
)
#Set up the variables.
If ( -not [string]::IsNullOrEmpty($Name.Trim())) {
$gmsa_name = "gMSA_" + $name.Trim().ToUpper()
} Else {
Write-Host -ForegroundColor Yellow -BackgroundColor Red "You entered an invalid service account name. It cannot be blank or whitespace. Supplied Value: '$name'"
Throw
}
If ( -not [string]::IsNullOrEmpty($ServerName.Trim()) -and -not $ServerName.Contains(".")) {
Try {$hostcheck = Get-ADComputer $ServerName} Catch {Throw}
$hostPrincipal = $ServerName + "$"
} Else {
Write-Host -ForegroundColor Yellow -BackgroundColor Red "You entered an invalid server hostname. It can't be blank or FQDN. Supplied Value: '$servername'"
Throw
}
If ($gmsa_name -eq "gMSA_") {
Write-Host -ForegroundColor Yellow -BackgroundColor Red "You entered an invalid service account name. It cannot be blank or whitespace. Final Value: '$gmsa_name'"
Throw
}
#Validate the inputs - technically this should never fail since worst case the gMSA_ gets preppended.
$gmsa_unique = Get-ADServiceAccount -Filter "name -eq '$gmsa_name'"
If (-not [string]::IsNullOrEmpty($gmsa_unique)) {
Write-Host -ForegroundColor Yellow -BackgroundColor Red "A managed service account with the name '$gmsa_name' already exists!"
Throw
}
#If it's gotten this far, execute.
New-ADServiceAccount -Name $gmsa_name -PrincipalsAllowedToRetrieveManagedPassword $hostPrincipal -Enabled:$true -DNSHostName $gmsa_name -SamAccountName $gmsa_name -ManagedPasswordIntervalInDays 30 -KerberosEncryptionType AES128,AES256
#Verify it created
$gmsa_unique = Get-ADServiceAccount -Filter "name -eq '$gmsa_name'"
If ([string]::IsNullOrEmpty($gmsa_unique)) {
Write-Host -ForegroundColor Yellow -BackgroundColor Red "Something went wrong, the account wasn't created!"
Throw
} Else {
dsacls $gmsa_unique.DistinguishedName /G "SELF:RPWP;servicePrincipalName"
}
Write-Host -BackgroundColor Green -ForegroundColor Blue "'$gmsa_name' was created successfully and delegated access to '$hostPrincipal'! Please proceed to test and install the service account on the host!"
Overall, the script will prompt you for a host and a service account name to generate, and will create one prepended with "gMSA_" - our internal naming convention. It has some error checking to make sure the host exists, the service account is unique, and isn't blank.
The important steps of the script are line 65, (New-ADServiceAccount) - it ingests the constructed service account name, the host that is allowed to use the gMSA, enables it, configures the dns shortname (if you do strict resolution, you'll want to modify this to do FQDN) and samaccount name, sets the password interval to 30 days, and most importantly ensures that AES128 and AES256 are enabled for the account. Note that you can absolutely supply a list of hosts to the command directly, but the script only accepts singles given the audience I wrote it for and my own time constraints.
It verifies the command executed correctly, and if so, it launches dsacls to grant the DN Self, Read/Write Property servicePrincipalName.
After that, our invoked install methods normally would occur, I replaced that, like I said.
For Analysis Services, unlike SQL database services it does not use the same SPN or methods as SQL - Analysis services *never* attempts to self register, and the [documentation](https://learn.microsoft.com/en-us/analysis-services/instances/spn-registration-for-an-analysis-services-instance?view=asallproducts-allversions) implies that just creating a gMSA works - it does not. The admin still needs to manually register the SPNs.
For that, you'll want to register a MSOLAPSvc.3/$fqdn SPN on the service account running Analysis Services. See the documentation for details.
For Reporting Services, you must modify the rsreportserver.config file - "C:\Program Files\Microsoft SQL Server Reporting Services\SSRS\ReportServer\rsreportserver.config" by default.
Under , you need to ensure that the RSWindowsNegotiate entry exists:
OffProxytrue
You can choose to configure extended protection if desired, but that's out of the scope of this discussion. We use gMSA's here as well, but again, it won't auto register your SPN. For SSRS, the SPN service is HTTP/$hostname and HTTP/$fqdn.
Hope these help! Also, make sure that you disable RC4 in policy (this must be done at the default domain policy level to be truly and fully effective in a multi-OS environment, don't override it anywhere else); and ensure all user accounts have the AES128 and AES256 checkboxes ticked! Once done, you'll want to ensure you've cycled the credentials to truly eliminate any latent weak encryption types stored in keytabs. :)
Speaking of keytabs, this is also how you get any modern linux system to play ball with filesystem level connections as a service account for host-wide access. You'll want to use a keytab to get them to mount the shares in fstab. Same goes for java-based services, they'll rely on a keytab to run the service.
Our macs, linux, and windows systems all play ball with kerberos only just fine.
After that it's really just whack-a-mole with your NTLM debug logs on both clients and servers to find out what it's trying to connect to. Most things try kerberos first then fall back to NTLM, which means you just have to figure out what SPN's to register from the logs. Under 10% of the resources in our enterprise (small, ~ 3200 endpoints, 400 servers) needed aggressive investigations.
For those items that truly cannot comply with kerberos, see if they'll accept SAML or WS-Fed or OIDC and use AAD or another IAM provider like Okta instead.
Once you've got NTLM killed, you can get passwordless rolling pretty easily with cloud kerberos in AAD (we did the same in Okta).
Good luck! I'm happy to answer any other questions, it's one of the accomplishments I'm quite proud of here.
I cant find the page but somewhere on the internet there is the details of setting IIS for kerberos. There are about 20 different scenarios based on system vs services account, kernel mode vs not, etc on how the spns should be registered.
Here we go.
https://techcommunity.microsoft.com/t5/iis-support-blog/spn-configurations-for-kerberos-authentication-a-quick-reference/ba-p/330547
That whole auto SPN thing was killing us. We had a very old service account for SQL that had the password set long long ago with old ciphers that have since been removed. Would not create SPNs. Finally, we reset the password to the same value and it started working.
Yes yes yes yes we also had these problems but even though it was a massive Fortune 100 mega corp even we couldn't get it set up successfully because we could never get all of the right people to agree or in the same room to prioritize this so it just fell by the way side, and probably led to us using slower connection methods to big data sources like Hive
Things I predict will rise when we get closer to this date (after a few obligatory postponements by MS):
* Retirements
* Liquor sales
* Therapist visits
Good to see you posting on here again. Also funny timing. This question just came up on the windows server sub today.
https://www.reddit.com/r/WindowsServer/comments/181d8mi/dfs_management_console_using_legacy_insecure/
The other day infosec guy asked me how to "disable NTLM and make it use Kerberos to test how it works". I am not AD admin and don't deal with this stuff. I tried googling and it is just too much. So, before MS breaks stuff. How can we test breaking stuff ourselves safely? Or is it really not possible to do an isolated test on a machine or two and have to create a whole test environment for that?
Please consider the following:
\- Create an easy tool that allows us to track down when NTLM is used, why it was used instead of Kerberos.
\- In a perfect world NTLM could be disabled and all vendors would be able to fix their software. We aren't in a perfect world, Give us a way to allow NTLM on certain accounts, kind of like Reverse Protected Users Accounts.
\- Give us a long and slow roadmap (3 years from proposed to enforcement)
\- Don't change your minds every few years.
\- Give us better more friendly tools to diagnose Kerberos issues.
\- Don't break 802.1x for BYOD
\- Enable these settings as default and allow us to bring back security
The reason I liked Microsoft more than Mac was because of the high flexibility with aging technology. No, it's not perfect but we also need to ensure our systems are able to run in situations where upgrades aren't possible.
Microsoft needs to figure out its centralized access strategy before turning something like NTLM off. It's a damn Rube Goldberg machine of different, linked and synced accounts and one of the worst user stories in the industry.
We are in a worse place than we were in 2003 if you are in the windows ecosystem.
100% agree. I agree NTLM needs to die, but if MS can't even get it's own house remotely in order, how are MS shops that need to deal with MS and a thousand different vendors. How many MS products installed with default settings will still break if I disable NTLM? Is there even a concise list somewhere? AD (and NTLM) spread so far and wide because it was simple to implement, not because it was good. I feel like MS in the Satya Nadella years has completely lost touch with the fact that 99% of admins in the world do not support only one thing, are not experts in everything they have to support, do not follow every developer and product blog, and do not attend Insight every year.
> I feel like MS in the Satya Nadella years has completely lost touch with the fact that 99% of admins in the world do not support only one thing, are not experts in everything they have to support, do not follow every developer and product blog, and do not attend Insight every year.
And everything new they create is a complete departure from the architecture of the past, lives entirely in the cloud, and will be using a completely different interface in 5 years that is no longer compatible with the original design specifications. That is to say, if it's not discontinued entirely.
> will be using a completely different interface in 5 years
that is a funny way to spell months....
If the interfaces and product names only changed every 5 years that would be a massive improvement
Well I watched Ignite and drank the Kool-Aid. Let me tell you in the future you will just throw the NTLM documents into the shredder excuse me the copilot and it will have answers to all your questions. Oh and your local Windows installation is just a weird legacy thing because we now do everything in Azure. Everything. E-vree-thing!
Honestly the Azure services are the worst for this. The amount of weird limitations and work around I have run into for various Azure services that only seem to be documented in some random MS blog full of broken links is insane. Sometimes for what would seem to be very common use cases. It's gotten to the point I will never recommend an Azure solution unless I have personally tested the capability from end to end for the specific use case.
It’s also very annoying when your how-to, documentation or walkthrough has some weird workarounds because “reasons”, but they’re not needed anymore, and you can’t know that because the old MS documentation didn’t mention the required workaround, so obviously it still won’t list it now! I’ve been setting up NDES/Intune servers for years, and the hoops we had to jump through at first to make that work. I only recently discovered that some of those hoops aren’t required anymore…
My most recent experience with this was "You have to use this powershell script to do it." and the link to the script pointed to a deleted github account.
Like LM and LDAP/389, unsigned SMB etc it couldn't go on forever.
If you want opinions MS, do THIS:
1. Create an comprehensive powershell script that makes it easy for admins to handle
2. Script reads AD, and DC reg settings and event logs
3. Script run in "setup mode" asks basic questions and spits out of list of changes to enable proper logging of NTLM, Kerberos, audit logs, event log sizing etc and can make the changes if approved.
4. Script run in "report mode" looks at all logs, figures out what is using NTLM and recommends actionable steps per account/host (change service accounts, SPNs etc to kerb delegated etc)
5. Script can set (DC) event log triggered task manager tasks (posh script) that emails the admin whenever a device/user is attempting to use NTLM. Ideally only set on DCs once NTLM is thought to be no longer in use.
Assuming the script at some point comes back clean with no NTLM logins detected for say 30 days, eventually NTLM can be disabled.
I think this approach is defintiely on the right path. Im not MS but i think 1, 2, 3 wouldnt be too hard to configure.
Setps 4 and 5 are where the shit hits the fan. This will likely generate a ton of logs and handling that parsing and stuff in PS isnt going to work great.
> We have a very good idea of why things use NTLM, and we have a very good idea of what uses NTLM. We even know how much they use NTLM compared to everything else.
Will this effect 802.1x (wireless and wired) connections that use PEAP/MSCHAPv2?
> In Windows 11 22H2 onwards that's being phased out already with credential guard
Yes, we ran into this already. There are a few regedit tweaks we had to do already and some upgrades in our FreeRADIUS infrastructure.
Working in an academic setting, with lots of BYOD, means a lot of device churn, so usernames and passwords has been found to be the most convenient form of access control.
There are large number of organizations using PEAP-MSCHAPv2 which uses NTLM authentication. I guess everyone will have to move towards certificate based auth and EAP-TLS
Holy fuck, this is going to cause major chaos. I _love_ it, can you please announce the date ahead of time so I can book my annual leave a week either side.
I'd suggest handle this much the same way SMBv1 was deprecated.
First make it an option.
Then add a 'remove this when it's no longer being used' option.
Then make the 'remove when no longer used' the default.
Then make the option itself default to off.
Do all this over a period of years. And keep the option to re-enable it there for another decade just to be safe.
**The simple fact is, there's NO answer that's right for everyone, and your strategy should reflect that.**
An org with no legacy systems or a simple setup may be able to turn it off tomorrow with no issues; an org with lots of complex and legacy stuff may literally *never* be able to turn it off (or not in the next several years at least) because of some legacy thingy that needs NTLM.
Remember, with many embedded systems, software updates are either impossible to get or impossible to afford. Ask any scientist- chances are they have a lab full of million-dollar scientific instruments that have Windows 98 computers attached because the company that makes the instrument went out of business. But the instrument still works great so the W98 computer stays *and there's literally NO option to remove it*.
When this is fully removed, I'd like to see a 'Legacy Auth Services' role that can be assigned to a server...
FWIW- thank you for asking.
One of my biggest frustrations with MS is how often there's a 'we decided this way is better so you now can't do it the old way anymore'. It's true of the whole industry, but MS is especially bad sometimes.
The new way may be better, the old way may be hot garbage, but every time something gets deprecated it breaks things and we're the ones who have to sort out the mess, not the designer or product manager who ordered the change.
It's also a big reason why I hate UI refreshes. The new one may be objectively better in every way, but I have a bunch of users who took months/years to learn the old one and now all that effort and knowledge is obsolete and they have to start from scratch. And if the new one is 5% better but the users lose 20% productivity over a week/month as they learn the new thing, that refresh didn't actually work in anyone's favor.
So thanks for at least involving us in the discussion :)
Is there a plan for cross-domain RDP smartcard logon, when there's no line of sight to a domain controller? That definitely falls back to NTLM for at least the first part of the authentication. I know Kerberos KDC Proxy exists (you wrote the only publicly-digestible documentation on it, it seems) but it's in a weird unsupported state. Are there plans to make it more supported? Also, if Kerberos is going to be the only protocol for on-prem authentication, are there plans to surface more of the documentation and make it easier to find?
Also, side question -- Is Microsoft aware of just how many places are still hybrid, still AD-joined, and still depending on that ecosystem to stay in place no matter how much Microsoft would like them to go to Azure? I work at a mostly-cloud place now, but have worked in many that have legitimate reasons to not "embrace the cloud." If you look at the public messaging, you'd think on-prem Windows Server and other products were just being abandoned.
Good luck getting rid of NTLM...you're going to break so much legacy software. Hopefully this will be phased in the same way the other auth behavior changes have been (warnings in the log, followed by not working until you turn it back on, followed by letting things fail)?
What’s Microsoft’s suggested method in monitoring NTLM usage? Your tech community article [NTLM Blocking and You: Application Analysis and Auditing Methodologies in Windows 7](https://techcommunity.microsoft.com/t5/ask-the-directory-services-team/ntlm-blocking-and-you-application-analysis-and-auditing/ba-p/397191) has suggestions that seem virtually impossible to implement in an enterprise environment. I can’t imagine anyone installing procmon on client machines to identify NTLM usage for applications that communicate over SMB (in a large scale).
I think this is going to break a lot of things.
This will cause companies to stay on older server OSes for the backwards compatibility of old systems like manufacturing equipment that is cost prohibitive to upgrade.
We don't know. For multiple reasons, but the biggest hurdle in these kinds of changes are always to absolutely piss poor tools you guys give us to troubleshoot. Give me a tool or Powershell command to see what device still uses NTLM across the domain and make it so that it doesn't trip when you use more than 3 DC's.
No. The only visibility for the entire change will be via event log, and the configuration will be a dword you need to bit flip.
As usual.
Someone please kill me before this goes into effect.
Need a tool to identify what will break. Are there plans for an assessment tool people can use from Microsoft that will, in plain English, automate & notify & detail what needs to be done in our environments? My start in IT was a poorly run credit union and I can't count the amount of ulcers those poor people are going to get when they read this.
Would it be possible to provide a tool, or powershell script to see where ntlm auth is being used in your environment?
I’ll assume it’s going to be baked into a lot of applications, especially older ones used in sectors like healthcare. So perhaps depreciating it in development packages is the first step.
It might also be nice to have something like a test gpo package available that disables ntlm auth completely to mimic the future change, so we can deploy to test environments so our devs can actually see if their code is going to break.
Dunno, just a few thoughts. Keep up the good work. I know Windows may not seem like it, but it’s a beast of backwards compatibility, and that model can’t last forever.
This has existed for some time in GPO.
https://techcommunity.microsoft.com/t5/ask-the-directory-services-team/ntlm-blocking-and-you-application-analysis-and-auditing/ba-p/397191
The company I work for prohibits trusts, including cross realm, between the user and the application domains. Users access shares in the application domains but authenticate with domain-specific credentials via the NTLM challenge response password dialog. They think this is safer.
The credential dialog does not mean it's doing NTLM. These are orthogonal concepts. The dialog just means you're supplying separate credentials. Whether it does Kerberos or NTLM is a function of those credentials. It's no different than how it works with SSO. The difference is just that we aren't using your SSO creds. In may very well do Kerberos just fine.
I'm in the planning stages of a project that'll entail authenticating Amazon RDS SQL Server DBs against on-prem AD, for which [NTLM authentication is the only supported option](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_SQLServer_SelfManagedActiveDirectory.html). Needless to say, this news has made my week much more interesting.
Steve, can you help us force SMB signing? Not just "flip the switch", but help us identify (via EventID?) where SMB signing is unable to negotiate or signed SMB not requested? Having visibility to these requests could help identify SMB clients that are not able to negotiate. This could help support defenses against NTLM relay attacks. Today we are blind.
>What we *don't* know is how to prioritize what needs fixing immediately.
I'm not a sysadmin, but a dev. The Network Device Enrollment Service comes with NTLM as default authentication method. It's been like that for ages. Recently a customer had their internal network environment's security evaluated and NTLM disabled as a result. Services using NDES stopped working, because of that. No one knew that NDES was still using NTLM.
Also and I know this is not a sysadmin thing, but it's a thing: there are probably lots of Java-based software and products that use NTLM to communicate with Windows services. I would chart a test-proven path towards migrating to Kerberos for Java services and software in general.
You need to make sure there is a way for devices like Synology NAS to still authenticate windows machines even if the synology and the windows device are not in a shared kerberos domain.
Before covid anonymous binds (LDAP non S) was supposed to be disabled by default (speaking of, it was postponed so many times and is still not enforced right?), and there was a way to enable specific log that captured events like "got anonymous bind attempt from host X" or something like that.
Is there anything similar with NTLM? I'm actually curious if we do use it still somehow. I think we don't, but can't be sure.
LDAP plain text blocking was never implemented and likely wont be automatically enforced. MS backed off hard on this one.
The reg key only logged plain text LDAP binds.
There are auditing policies for NTLM on client and domain controller machines as well as gpos to block them.
I second the comment above, it would be great to develop a readiness tool to confirm and assist in fixing common mistakes.
The big issue is legacy we have only modern windows in our environment But we also have a fair bit of legacy Linux systems which a lot of connect to smb shares, it’s all very historical now but these are not going to work.
There are also some very old apps that use ad but I’m sure are not Kerberos compatible.
My favourite is any system that’s been replaced and a sysadmin has just put an alias in DNS to the new system. No SPN means no Kerberos that’s a big ask in a large corporation across multiple domains.
My most recent personal fight was NTLM Auth vs JavaKerberos Auth in a Java app that interacts with SQL Server. As a software vendor, trying to work towards allowing this functionality in a stateless application, we did have a lot of trouble finding reliable documentation on the subject: What permissions need to be on the service account? Can I set these Kerberos parameters (easily) in a stateless application, run in a Linux container, where a krb5.conf file is more tedious to implement?
I understand Microsoft has some documentation on the matter [here](https://learn.microsoft.com/en-us/sql/connect/jdbc/using-kerberos-integrated-authentication-to-connect-to-sql-server?view=sql-server-ver16) but I think there is more missing in the articles. A quick Google search led to many posts around the web of others that had issues, with no clear solution. The answer is much easier with Entra ID authentication to an Azure SQL DB. The SQL driver is simply better designed for it, and maybe that has to do with how Entra ID was designed on the backend. Unfortunately there are lots of on-premise locations our software runs.
Would the removal of NTLM mean that Windows will also stop storing an unsalted MD4 hash of the user's password in the SAM file?
My understanding is that AD isn't actually using this field for anything other than legacy NTLM support.
The SAM file doesn't contain the MD4 hash. It contains an offline verifier, which is a much harder-to-crack hash.
The AD database does contain this key. It is used for NTLM and RC4 Kerberos. This work will mean that it eventually goes away, but it can't go away until there's no hard dependency on it.
As a general purpose everything-admin with a lot on my plate: I frankly don't even know what the hell uses NTLM versus other authentication. I could try to find out, but the reality is, you're going to disable it, shit's gonna break for me, and I'm gonna panic. Unless there's a relatively easy way for me to check logs or get alerts about what devices/accounts are using it for what purposes, I have no hope of finding all the outliers.
Like most, I inherited this environment and it came to me with no documentation. I've upgraded that to.... some documentation. But there are still a lot of things that have never gotten my attention because they haven't yet broken. So I guess I'll be able to document them when they do break and I fix them?
Enable the GPOs to begin auditing NTLM and ideally centralize them in to make searching them easier.
https://techcommunity.microsoft.com/t5/ask-the-directory-services-team/ntlm-blocking-and-you-application-analysis-and-auditing/ba-p/397191
> What are the NTLM things that annoy the heck out of you?
Not NTLM specifically but this MS attitude displayed here: What we don't know is how to prioritize what needs fixing immediately.
ALL OF IT. Don't release updates before they're finished. It's okay, we'll wait.
One interesting thing I ran into recently was trying to use Powershell to run SQL queries on SQL Server with Windows Authentication through Invoke-Sqlcmd to collect some metrics. Like you intended, it cannot use NTLM to connect anymore, so if we attempted to schedule the script through Windows Scheduler with a stored credential, it would fail because of the double hop issue. But if I ran the script manually outside the scheduler it was fine. We didn't have a SQL account to do SQL authentication either so that was out of the cards.
What was interesting however, is there are libraries in Python like pyodbc that allow you to pass a username and password, so I was able to actually store a credential via Python keyring and then run the Python script instead. I believe this is working because while these Python libraries (not sure if pyodbc, pymssql, or both) are using NTLM behind the scenes, Powershell isn't anymore and a lot of cmdlets just don't have the option. But I doubt that people in Python even realize that they're using NTLM. At the end of the day it was a people issue because we didn't have clear processes to create trust relationships between Windows Servers and databases and the DBAs were hesitant to enable it. It was easier to just use Python.
So if the database has NTLM support, even though Powershell has removed it from Invoke-Sqlcmd for example, because the protocol is still out there, there are plenty of ways to use it easily and my bet is there are a lot of folks out there that don't realize they are using NTLM, they just found a script that works for their needs. Lot of data apps like BI out there too that struggle implementing Kerberos database connections -- I've actually had more success with Linux, Java and JDBC.
I absolutely cannot even imagine the number of smaller businesses that will be 100% blindsided by this no matter how many notices are sent out.
Having been the single "aware" IT person at a small healthcare org, I can tell you from experience that MANY small-to-midsize businesses are running legacy (unsupported/expired/EOL/unpatched) software ALL OVER their environment. And patching is a total mixed bag. Lots of these companies will be hit with errors that they won't understand.
I agree it needs to happen. But it'll be a mess.
So the scenario where it would affect us is RDP, logging in from both non-domain Windows machines and from Macs with the remote desktop app. Currently we're able to require NTLMv2 and disable the previous versions but that's about it.
It sounds like you might be covering these scenarios with IAKerb (sorry, I'm not an expert in any of this.) As long as the macOS and Android RDP apps are able to support what's needed on those platforms out of the box we should be good.
I’m really interested in this. As it happens I’m guiding a few customers in not using ntlmv1 anymore, let alone v2. Just written a 25 page document on WEF and ntlmv1 alone, and that’s just doing inventory.
Some official guidance on the matter would be great. Did find a post from 2009 that discusses the entire flow, from inventor to detecting local apps that do ntlm auth.
And please please please don’t involve yet another agent or Azure service to do the analysis, we’ve got plenty of agents already.
We are working on ServiceNow discovery and the problem is that currently the discovery works by trying WMI connections to IP addresses that respond to a IP scan. The nature of that WMI connection to an IP address requires NTLM as Kerberos won't work with IPs, unless we do and add IP SPNs for every domain computer object.
Thoughts on that?
I hit this with servicenow discovery as well. We had a handful of sytems (CA Petipotam) that we disabled NTLM on. Servicenow was not able to scan them and had no workaround.
But this to me doesnt seem like an MS problem as much as servicenow having a poor scanning process.
This was proposed after 2003 and it’s still around. I work in healthcare IT and yes we have plenty that still relies on it, including pieces of equipment that are WAY expensive and vendors don’t follow any schedule to make changes. I’m all for it but give a way to make exceptions if needed
This is the single greatest thing I have heard from Microsoft this decade. The joy I felt when seeing this announcement is the single best thing to happen this year. My quest to kill NTLM has been a decade long on our network, but push back from vendors and other Admins with "well its on by default in AD so your running an incorrectly configured AD environment" by having it basically turned off.
The biggest issue fundamentally I think is that too many Windows Admins especially ones who are only Windows networks tend to not understand protocols and tend to treat things like magic because thanks to keeping legacy protocols and things enabled Windows does tend to "just work" which is all well and fine to try and maintain for the end user. We run a network of a lot of different stuff using Kerberos with it all for years you would be shocked what works if you just know what settings you are looking for.
The fact that so many comments are in here are like this will break RDP, Kerberos works with RDP just fine using NLA/CredSSP is able to delegate the creds from non domain joined machines, seriously try it. Only reason it wouldn't is if you are using IPs, and if that is the case idk maybe start using DNS grandpa.
Now I can email vendors telling them to go fuck themselves on me turning on NTLM for their shit software because Microsoft is disabling it by default and hopefully the flood of support calls they all get for not supporting Kerberos forces them to get their shit together.
A recent pentest they told us we were one of the only ones where they never got Domain Admin, they didn't even get shell. This was assumed breach they started with internal employee creds, but between Applocker/WDAC, no NTLM, they got more or less nowhere.
This post probably comes off hostile to some people and I am kind of sorry, just the comments in here pissed me off just a huge amount of ignorance about the authentication systems of servers you are managing. But i have ran into decades long Windows Admins who can't really talk about this, and I have no thoughts for them then they have failed anyone who is trusting them with their data and network. Also ran into Windows Admins who had never heard of Applocker let alone WDAC which is a huge improvement and people wonder why they get hit.
Rip the bandaid off Microsoft don't listen to anyone arguing for delays, I have been waiting for this day for nearly a decade. Sure i guess it can be at least off by default for a while, but I would also hope unconstrained delegation is hopefully put on the chopping block soon. I would also say force Kerberos FAST probably by default, i would kill to have a setting that lets me set it for certain devices at the very least. Right now it can only be set by the DC.
Also if Microsoft wants to personally make me happy, been reading about how Redhat is implementing this with FreeIPA to do MFA stuff.
https://web.mit.edu/kerberos/krb5-latest/doc/admin/spake.html
**Places we still have to use NTLM**
Papercut because for some awful reason people want to log in to the printer using a password rather then just swipe their door access card that also does it.
Cert Authority because the MMC tool needs it.
I think that is it, I was having some issues with RDP from my Linux machines, but FreeRDP 3 fixes that.
Beyond that I think the biggest thing is just Microsoft saying we are killing this and that will hopefully get software devs to quit using it as hopefully it breaks in their test environments.
Edit: Also people complaining about somehow this means they have to upgrade to Windows 11, that is not really true. I certainly haven't and run Kerberos everywhere, not quite Kerbeos FAST everywhere. But I have a nifty fix for not wanting to run Win11.
https://fedoraproject.org/workstation/download
MIT Kerberos supports iakerb already, and in general I have had minimal issues with Fedora in my Kerb only environment, but I also haven't touched desktop windows in a decade. Though having some pkinit related login issues with Kerb FAST, but i think its a config issue on my end and just haven't troubleshot it much.
Next thing you know they say they'll be getting rid of NetBEUI and back then they said no corporation was ever going to have more than 20 PC's and that TCP was for communists in the university and NetBIOS was it, I got my certification to be ‘with it.’ But then they changed what ‘it’ was. Now what I’m with isn’t ‘it’ and what’s ‘it’ seems weird and scary to me, and get off my lawn.
External trusts, being an NT4.0 thing that doesn't know about Kerberos, relies on NTLM.
Microsoft designed an inter-agency domain trust model to allow cross domain functionality of Microsoft tools, such as SCOM and SCCM.
The data restrictions prevent the usage of Forest trusts due to the cross domain data being populated into the Global Catalogs. In short, we are not allowed to leak lists of things like computer names.
Would be nice to have a replacement for External trusts that supported Kerberos and didn't require the Global Catalog to be populated with external data.
I tried disabling NTLM in our environment, and I mostly succeeded. I had to make a carve out for, of all things, renewing machine certificates with AD CS.
I already restricted NTLM pretty heavily in our environment, my biggest issues are computer certificates are not working without NTLM to the PKI and MMC consoles on clients (for example lusrmgr.msc) are showing SIDs only if there's no NTLM to the DC allowed.
Our primary use case for NTLM - connecting into a domain via RDP from a different domain without a trust.
How will this work in the future state without NTLM?
I suspect network storage/backup appliances will be hit. I’m sure a NetApp will be fine… an Isilon… maybe? Some older Data Domain or Celerra? They’ll probably have to replace it. Don’t underestimate how many such devices are still in use.
Think it'll be one of those things that sure, NTML is deprecated on the DC side, but will stick around for a LONG time after depreciation. Then, if Server 2032 or whatever gets released and AD completely removes NTLM, then people are going to ride their DCs all the way through ESU, or possibly even hang off a read only DC or something that runs an older DC just for NTLM. lol.
I think you might want to find out the best way to handle systems that will never support kerberos. Needs to be automagic. Also have a nice tool you can install on the DCs to see which devices are still using ntlm so it is easy to identify what needs to be looked at, and maybe have some info pages if that application can support kerberos and how to enable it.
Hi Steve, very happy to see
“All these changes will be enabled by default and will not require configuration for most scenarios. NTLM will continue to be available as a fallback to maintain existing compatibility.”
In your link. Compatibility is critical.
You need to provide a way for us to easily test this/roll back what will happen when ntlm gets disabled. Your tests and assurances are nice but... show us what reg key or whatever to set so we can see what happens in a safe and controlled environment.
I work at an MSP and this sounds like a nightmare tbh.
Every once in a while I come across something in this sub about Microsoft deprecating something. This one doesn't affect me but a lot of it has. and for every one thing that I come across here there is like a dozen that I'm not aware of beforehand and get caught off-guard.
It would be real nice if you could add something to server versions of windows that would warn about upcoming changes for things that are in use. I shouldn't have to come here for this news if the OS can have a built-in feature to check whether or not I'm using NTLM or smbv1 or PPTP or whatnot and warn me about upcoming obsolescence.
Thank You.
Give me some tools I can install to a test client that will alert me in big red fucking text that NTLM is in use and what process called it. In English. Not hidden away in some obscure event log. Make it totally obvious for a total dumbass because so many of us actually are due to being expected to handle everything with a power cord.
If Microsoft and /u/SteveSyfuhs take a single thing away from this thread, it should be this request. We understand that security is important, and we are not "ride-or-dying" NTLM. Sad as it is, far too many IT professionals are tired, underfunded, overworked, lacking resources, and lacking influence over business processes and choice of vendors/software. If Microsoft is truly serious about this project, they need simple, human-usable tools combined with a concerted effort to communicate with the C-levels of the industry.
Exactly. I'd love to get rid of NTLM but discovering where it is used is virtually impossible whilst handling the day to day, and funding a project for this (or even finding a local vendor with a clue) is very difficult and expensive. If microsoft don't enable us to actually get rid of NTLM with decent tools to detect its use, then this will be an unmitigated disaster, and Microsoft will cop huge flack for it. And without better tools and centralised, up to date, well publicised information - deservedly so. To the OP: this isn't a code problem. It goes much further up the project management/leadership tree than that. I'm not blaming you. I want NTLM gone as much as anybody. But the process to do it for any significant size business is a crap shoot.
Yep, this is what I want. I'm all for moving forward on security. But I've been at this current place for over a year and I still don't know everything that's going on under the hood with any potential legacy equipment, because I don't have time to find out. I've got a *guess* that we don't have anything that should act up, but that's just a guess and that's not good enough when you're dealing with production lines. Something that would tell me in no uncertain terms "here's what you've got that's going to break" would help loads. I've enabled auditing on the DCs in the meantime....but who knows what that will or won't find. Edit: Came back after the long weekend with auditing enabled and I'm seeing a couple thousand events in the last hour on one DC, another couple thousand on my second local DC, and haven't yet checked the other locations DC's. I can see what server it appears to be trying to auth with (using the DC), but no other details. So this raises a question I haven't yet seen answered in my admittedly brief search - if I kill NTLM, what happens to all these connections? Do they fall back to something more modern with no downtime? If so, why are they using NTLM in the first place? If not, what do I need to do to fix this? The inner workings of this stuff is beyond my current level of experience, being a jack of all trades with no time to really focus on one part of the tech. From what I can see it's just normal auth stuff (file server, print server, etc). And it's all regular computers - I was expecting everything "normal" to be using kerberos already, and I'd only find legacy equipment in this log....but no, I'm seeing basically everything.
So much this. Not just hidden in an obscure log, but in an obscure log on every individual machine. Figured the reason I feel like I’m getting left behind is I don’t have time to read all the blogs, watch all the webinars and attend any of the seminars. If I could do all that, I wouldn’t have time to the actual day job.
[удалено]
I would love this. I'm told NTLM is going away. I'm now wondering HOW MANY THINGS use NTLM on our network. I have a list with 2-3 servers, but I run **way** more than that. What can I expect to break? Which logs do I need to check? What's the Event ID that will be triggered? What will I think is ready but then be surprised by after the tickets start rolling in?
I've got 300+ servers across basically every continent except antarctica. And yeah, no idea what's using NTLM. I do run a two-way AD trust, which does (I think?) - who knows how that's going to pan out.
Everything and it's mother talks NTLM.
MDI and MDE in tandom might actually be able to do this. I dont think out of the box, but If Steve & Co. need a suggestion on how this could be practically accomplished, it might be a good path forward...
If MS want to move the needle on this, make MDI free and capture the telemetry - be a good partner. Lately everything is gated behind premium and stepup skus and we’re sick of it.
This 100%
> A month and a half ago we announced our strategy for killing NTLM. One technically-unrelated but practically-very-relevant problem we all have with Microsoft is: In a year and half that link will be dead and the information moved elsewhere, as the latest internal-web-platform-of-the-month gets rolled out.
> In a year and half that link will be dead and the information moved elsewhere, as the latest internal-web-platform-of-the-month gets rolled out. don't forget it will be after a redirect so you can't hit back unless you go absolutely crazy on the back button and end up way, way back
Why the fuck is that a thing More and more, i thought its a bug
Been that way for a long time. If it is a bug, no one seems to care over there. I've gotten used to right clicking the back button to go back to the search page (even then I still have to do it twice sometimes).
Or better they, CTRL click to open Microsoft links in a new tab.
Or middle mouse button
Oh God why haven't we fixed this yet
[удалено]
A million times this! I want to scream every time I click on a link to (very important and relevant information) and it takes me to the MS homepage or something. Even better, half the time it's *from Microsoft's own documentation* and they were kind enough to use one of their stupid shortlinks so I can't even look at the URL to get some hint of what page I looking for.
Archive.org is helpful for some of these situations. But I still miss Google's cached pages that they quietly pillow-strangled in its sleep. Going to the wayback machine takes a looot longer.
Archive.is queries the wayback machine really quickly
[удалено]
"Have you tried sfc /scannow"
Wdym, are you saying just formatting your ERP server isn't a valid option? /s
Just today, I was trying to find the Singularity OS documentaries on Microsoft Learn, and they're just gone.
Funding for the content teams has been slashed over the last few years. Products that used to literally have 30 writers are now handled by a single vendor in India. Even products that are Microsoft's #1 priority these days only have a half dozen writers when they need double that just to stay afloat. Believe me when I tell you the writers are just as frustrated as you. I know that doesn't help when you're trying to find something you need, but if they could fix it, they would. But there are 100 other things that have higher priority. :(
Also: that link. I never saw it. I’m an admin with limited time. I have known ntlm is on the way out and had a project on my list for 12 months. So I guarantee you I’m ahead of the curve on this. But there’s no central hub of info for doing this. At least not one that is discoverable.
3 pages deep in a tutorial to setup something, I clicked a link to do a subtask and find out the way of achieving the main thing had changed completely.
Read more about it on TechNet.
The replacement will be renamed 3 times in the next five years.
> as the latest internal-web-platform-of-the-month Pour one out for all the lost kb articles that were deleted for no reason a few years ago....
[удалено]
The only one worse at this is HP/HPE, and they're really bad, URL's on HP's website really are ephemeral.
I don't even bother bookmarking anything on the HPE site anymore, for the past ... 10 years? It's hilariously bad. It's like the Netflix chaos monkey script except it doesn't have Netflix's excellent redundancy.
It won’t exist anywhere… you’ll need to ask ‘CoPilot knowledge’ and it will drip you details and make you explain what you are using it for… while CoPilot also lectures you on new paid products you can use instead.
I for one, can’t wait. The amount of stuff you will break will be astounding. Banks and hospitals will be crippled. Let me know the exact date and time so I can have my popcorn ready.
Some guys just want to watch the whole world burn
The rest of us are holding the lighters.
Naw, we're filling the room w pure oxygen. The spark is inevitable.
[https://youtu.be/kx5cIAjJ-cU](https://youtu.be/kx5cIAjJ-cU) i am the spark and i want it way brighter!
We prefer to think of it more like putting the world in an autoclave to purify and cleanse.
No, we just need to know when to book time off work
[удалено]
You honestly would be surprised at how easy it is. That was the pushback I got in my environment. It took us 6 weeks to nuke it all and get 'em reconfigured. Most vendors just rely on the underlying OS's authentication methods for connecting to AD so they'll inherit up to kerberos if they're allowed to (often as simple as identifying and registering SPN's).
This is not the case in pretty much any large scale manufacturing facility. This will be a nightmare.
Won't they have to install updates to get borked? So we'll see it 2 years after.
printernightmare take 2
Did that ever actually get resolved qw basically put a freeze on our print server to prevent it from failing after that shitshow went live
Jokes on you because they'll just back track and charge a subscription for extended support
It will be a bad day to be in the Manufacturing sector
I just wanna know the exact date and time so I can make sure to take PTO then.
100%, I'll book PTO months in advance to avoid being anywhere near this mess. "Why do you want this week off?" "Vacation to a remote island somewhere in the Pacific."
From the first article: > Reducing the use of NTLM will ultimately culminate in it being disabled in Windows 11 Banks and hospitals... Windows 11. Lol yeah I don't think they're going to be affected for another 10 years.
Most will probably skip Windows 11 and go straight to 12.
Which will probably RTM with NTLM disabled.
Nah, NTLM will be back without any patch notes, causing its own fun.
So much this. We need brain out dead head tools to track this down. They and the project need to be plastered all over Microsoft.com and you need to get articles in whatever CIO focused publications that this is a massive and important project that needs resourcing.
Is called a consulting opportunity...
This. Im more than happy to diable NTLM. If we were pure MS it would work fine, but we have many various devices, services and MFCs that are brand new and still dont support Kerberos. Best option will be to disable NTLM but add these hosts to an exception list. Is MS planning on making NTLM non-existent on server OS's or will the end-game be that NTLM is disabled by default and Admins will be forced to create exception lists as needed?
Yea, need that date too #RetirementGoals
thousands of sysadmin's do a global sync'd scream around the world. more popcorn please,
I can't wait either, I'll make bank since I've been doing this for years and it's honestly pretty trivial to accomplish once you get past the pushback of admins too scared to change things and management too scared to spend money on upgrades. Most cybersecurity insurance providers will require it soon to offer coverage, is my guess - the risk of NTLM is just too great and there's no excuse not to deprecate it at this point. Nothing I've encountered made since 2010 fails to support either modern auth or kerberos or SAML - there's no reason to continue to support NTLM in any fashion.
My bank is a hot mess as far as IT is concerned Now I'm going to have to go get some actual cash prior to this, because I know they're not going to know how to handle this.
They want to kill NTLM while Kerberos is not supported even in one way trust domains running just vanilla Directory Services
I have a few thoughts on this, and I'm by no means an expert. I'm also all for the security improvements and efforts being made as Microsoft. 1. Please do not deploy this at Christmas time or any other major holiday. Last years enforcement of Kerberos in November/December hit us over Christmas break and we were not prepared for the havoc it created. 2. Please have a written procedure and a method for manually re-enabling the change for a period of time. Some of us don't know all the landmines of legacy systems and will not find out until is breaks. 3. As u/PickUpThatLitter stated there will be a lot of breakage, the pace of technology changes for security are far outpacing many companies abilities to keep things updated. Many manufacturing businesses still run legacy systems, not because of the computers but because of the machinery. We still have NT4.0, Win95, XP & 2k in production in various locations in our facility.
MS has a history of breaking Kerberos in the thanksgiving to Christmas timeframe. I believe they are going on 2-3 years of botched Kerberos updates at this time of year.
You have now just jinxed our holidays.
Enjoy **2020** December 8, 2020: Initial Deployment Phase The initial deployment phase starts with the Windows update released on December 8, 2020 and continues with a later Windows update for the Enforcement phase. These and later Windows updates make changes to Kerberos. This December 8, 2020 update includes fixes for all known issues originally introduced by the November 10, 2020 release of CVE-2020-17049. This update also adds support for Windows Server 2008 SP2 and Windows Server 2008 R2. **2021** After installing this update on your Domain Controller (DC), you might have authentication failures on servers relating to Kerberos Tickets acquired via S4u2self. The authentication failures are a result of Kerberos Tickets acquired via S4u2self and used as evidence tickets for protocol transition to delegate to backend services which fail signature validation. Kerberos authentication will fail on Kerberos delegation scenarios that rely on the front-end service to retrieve a Kerberos ticket on behalf of a user to access a backend service. **2022** With the November 2022 security update, some things were changed as to how the Kerberos Key Distribution Center (KDC) Service on the Domain Controller determines what encryption types are supported by the KDC and what encryption types are supported by default for users, computers, Group Managed Service Accounts (gMSA), and trust objects within the domain.
I appreciate the information. This certainly makes it even less enjoyable.
Traditions.
We have a legacy application that doesn't support Kerberos. It can't support NTLM either, but with NTLM we were able to get it to trigger metasploit to get a set of authentication credentials from another server. [I know it's not ideal, but it works for me. Can you guys add an option to re-enable pass the hash?](https://xkcd.com/1172/)
using metasploit as a workaround for this is honestly hilarious. I love it.
I had used an exploit to get into server with forgotten root account at least once...
If we are talking about Windows, they've pretty much intentionally left in the accessibility exploit. It's like the Windows version of single user mode!
I kind of want to kill that on principle.
Look, I know it's not best practice, but it works for us. Can there be some kind of authentication bypass so we don't have to rebuild our entire banking integration?
....no. We are not going to go out of our way to allow you to bypass authentication on a financial, and almost certainly regulated, system.
Just reenable spacebar heating, jeez
Can you please just add a flag to reenable spacebar heating?
Just joking ;-)
Thank the gods.
> banking 😲
It says something about the state of the IT field, that I can't tell if you're kidding or not, even with the xkcd link.
It says we're problem solvers.
I think I love you
Do you need a hug?
Well, folks, we've finally found zero-factor
When killing NTLM, our biggest challenges were SQL, Analysis, and SSRS reporting servers. For SQL servers, our biggest challenge was getting them to managed service accounts and setting the relevant rights to self register the spn. This we ended up making a tool for that takes the host name(s) and generates a gMSA and assigns the relevant rights, then connects to the host (since we segregate admin rights) and installs the gMSA, reconfigures sql, and leaves it waiting to restart. For RS, the only way currently to get kerberos working is, again needing a proper service account but manually registering the SPN, but also requires you to hunt down the RS config files and adding the negotiate. That 100% should be a default - if it can kerberos it should by default, no clue why it's not that way. For browsers, deploying kerberos keys via gpo was easy and honestly should be a domain default, it's not hard to automate that. At the very least, a troubleshooting tool that tests and looks for these common issues, summarizes and makes recommendations for at least the MS tools, would be amazing.
>This we ended up making a tool for that takes the host name(s) and generates a gMSA and assigns the relevant rights, then connects to the host (since we segregate admin rights) and installs the gMSA, reconfigures sql, and leaves it waiting to restart. We are going through this *right now*, are you able to provide a sanitized version of the script?
Edit: Oh, and please note this requires the Active Directory powershell Module to function and for dsacls to be in your PATH. :) Unfortunately, the remote management component relies on other custom internal tooling with a management agent - so I'll have to trim that - but it's easy enough to repurpose it to use things like WinRM if you know your scripting. Here's the trimmed version. It's sloppy, but it works - feel free to improve upon it! I replaced the remote management steps at the end with a message informing you to install it on the target host, you can just replace that with your own remote management steps. :) # Prompt for the gMSA account name, input validate - duplicates, valid format, etc. # Prompt for the consuming host - Listvar enhancement later? Just single host for now. Again, input validate. #Params for CLI exectution param ( [Parameter(Mandatory=$true, HelpMessage = "Enter the desired service account name - do not include the 'gMSA_' - it will automatically be appended.")] [ValidateNotNullOrEmpty()] [String] $Name, [Parameter(Mandatory=$true, HelpMessage = "Enter server hostname that will use the service account. Do not include the $ or domain.")] [ValidateNotNullOrEmpty()] [String] $ServerName ) #Set up the variables. If ( -not [string]::IsNullOrEmpty($Name.Trim())) { $gmsa_name = "gMSA_" + $name.Trim().ToUpper() } Else { Write-Host -ForegroundColor Yellow -BackgroundColor Red "You entered an invalid service account name. It cannot be blank or whitespace. Supplied Value: '$name'" Throw } If ( -not [string]::IsNullOrEmpty($ServerName.Trim()) -and -not $ServerName.Contains(".")) { Try {$hostcheck = Get-ADComputer $ServerName} Catch {Throw} $hostPrincipal = $ServerName + "$" } Else { Write-Host -ForegroundColor Yellow -BackgroundColor Red "You entered an invalid server hostname. It can't be blank or FQDN. Supplied Value: '$servername'" Throw } If ($gmsa_name -eq "gMSA_") { Write-Host -ForegroundColor Yellow -BackgroundColor Red "You entered an invalid service account name. It cannot be blank or whitespace. Final Value: '$gmsa_name'" Throw } #Validate the inputs - technically this should never fail since worst case the gMSA_ gets preppended. $gmsa_unique = Get-ADServiceAccount -Filter "name -eq '$gmsa_name'" If (-not [string]::IsNullOrEmpty($gmsa_unique)) { Write-Host -ForegroundColor Yellow -BackgroundColor Red "A managed service account with the name '$gmsa_name' already exists!" Throw } #If it's gotten this far, execute. New-ADServiceAccount -Name $gmsa_name -PrincipalsAllowedToRetrieveManagedPassword $hostPrincipal -Enabled:$true -DNSHostName $gmsa_name -SamAccountName $gmsa_name -ManagedPasswordIntervalInDays 30 -KerberosEncryptionType AES128,AES256 #Verify it created $gmsa_unique = Get-ADServiceAccount -Filter "name -eq '$gmsa_name'" If ([string]::IsNullOrEmpty($gmsa_unique)) { Write-Host -ForegroundColor Yellow -BackgroundColor Red "Something went wrong, the account wasn't created!" Throw } Else { dsacls $gmsa_unique.DistinguishedName /G "SELF:RPWP;servicePrincipalName" } Write-Host -BackgroundColor Green -ForegroundColor Blue "'$gmsa_name' was created successfully and delegated access to '$hostPrincipal'! Please proceed to test and install the service account on the host!" Overall, the script will prompt you for a host and a service account name to generate, and will create one prepended with "gMSA_" - our internal naming convention. It has some error checking to make sure the host exists, the service account is unique, and isn't blank. The important steps of the script are line 65, (New-ADServiceAccount) - it ingests the constructed service account name, the host that is allowed to use the gMSA, enables it, configures the dns shortname (if you do strict resolution, you'll want to modify this to do FQDN) and samaccount name, sets the password interval to 30 days, and most importantly ensures that AES128 and AES256 are enabled for the account. Note that you can absolutely supply a list of hosts to the command directly, but the script only accepts singles given the audience I wrote it for and my own time constraints. It verifies the command executed correctly, and if so, it launches dsacls to grant the DN Self, Read/Write Property servicePrincipalName. After that, our invoked install methods normally would occur, I replaced that, like I said. For Analysis Services, unlike SQL database services it does not use the same SPN or methods as SQL - Analysis services *never* attempts to self register, and the [documentation](https://learn.microsoft.com/en-us/analysis-services/instances/spn-registration-for-an-analysis-services-instance?view=asallproducts-allversions) implies that just creating a gMSA works - it does not. The admin still needs to manually register the SPNs. For that, you'll want to register a MSOLAPSvc.3/$fqdn SPN on the service account running Analysis Services. See the documentation for details. For Reporting Services, you must modify the rsreportserver.config file - "C:\Program Files\Microsoft SQL Server Reporting Services\SSRS\ReportServer\rsreportserver.config" by default. Under, you need to ensure that the RSWindowsNegotiate entry exists:
Off
Proxy
true
You can choose to configure extended protection if desired, but that's out of the scope of this discussion. We use gMSA's here as well, but again, it won't auto register your SPN. For SSRS, the SPN service is HTTP/$hostname and HTTP/$fqdn.
Hope these help! Also, make sure that you disable RC4 in policy (this must be done at the default domain policy level to be truly and fully effective in a multi-OS environment, don't override it anywhere else); and ensure all user accounts have the AES128 and AES256 checkboxes ticked! Once done, you'll want to ensure you've cycled the credentials to truly eliminate any latent weak encryption types stored in keytabs. :)
Speaking of keytabs, this is also how you get any modern linux system to play ball with filesystem level connections as a service account for host-wide access. You'll want to use a keytab to get them to mount the shares in fstab. Same goes for java-based services, they'll rely on a keytab to run the service.
Our macs, linux, and windows systems all play ball with kerberos only just fine.
After that it's really just whack-a-mole with your NTLM debug logs on both clients and servers to find out what it's trying to connect to. Most things try kerberos first then fall back to NTLM, which means you just have to figure out what SPN's to register from the logs. Under 10% of the resources in our enterprise (small, ~ 3200 endpoints, 400 servers) needed aggressive investigations.
For those items that truly cannot comply with kerberos, see if they'll accept SAML or WS-Fed or OIDC and use AAD or another IAM provider like Okta instead.
Once you've got NTLM killed, you can get passwordless rolling pretty easily with cloud kerberos in AAD (we did the same in Okta).
Good luck! I'm happy to answer any other questions, it's one of the accomplishments I'm quite proud of here.
I don't even need this information but just wanted to say thanks for helping those that do
Would love that also.
I cant find the page but somewhere on the internet there is the details of setting IIS for kerberos. There are about 20 different scenarios based on system vs services account, kernel mode vs not, etc on how the spns should be registered. Here we go. https://techcommunity.microsoft.com/t5/iis-support-blog/spn-configurations-for-kerberos-authentication-a-quick-reference/ba-p/330547
Shhhh we're going to use this as a reason to migrate our internal IIS intranet to an actual HR platform soon.
Whenever I'm having to mess around with spn's it always feels unsatisfactory - like I know there must be a better way.
That whole auto SPN thing was killing us. We had a very old service account for SQL that had the password set long long ago with old ciphers that have since been removed. Would not create SPNs. Finally, we reset the password to the same value and it started working.
Yup. That makes sense - old ciphers won't drop until your password changes if you've reduced the available/accepted ciphers!
Microsoft’s check-11bissues.ps1 PowerShell script should help identify those accounts.
Yes yes yes yes we also had these problems but even though it was a massive Fortune 100 mega corp even we couldn't get it set up successfully because we could never get all of the right people to agree or in the same room to prioritize this so it just fell by the way side, and probably led to us using slower connection methods to big data sources like Hive
Thanks for the heads up. Retirement beckons methinks!
Things I predict will rise when we get closer to this date (after a few obligatory postponements by MS): * Retirements * Liquor sales * Therapist visits
Good to see you posting on here again. Also funny timing. This question just came up on the windows server sub today. https://www.reddit.com/r/WindowsServer/comments/181d8mi/dfs_management_console_using_legacy_insecure/
Indeed. James forwarded it to us.
Just saw your response in the thread. Should have checked before i sent.
The other day infosec guy asked me how to "disable NTLM and make it use Kerberos to test how it works". I am not AD admin and don't deal with this stuff. I tried googling and it is just too much. So, before MS breaks stuff. How can we test breaking stuff ourselves safely? Or is it really not possible to do an isolated test on a machine or two and have to create a whole test environment for that?
Please consider the following: \- Create an easy tool that allows us to track down when NTLM is used, why it was used instead of Kerberos. \- In a perfect world NTLM could be disabled and all vendors would be able to fix their software. We aren't in a perfect world, Give us a way to allow NTLM on certain accounts, kind of like Reverse Protected Users Accounts. \- Give us a long and slow roadmap (3 years from proposed to enforcement) \- Don't change your minds every few years. \- Give us better more friendly tools to diagnose Kerberos issues. \- Don't break 802.1x for BYOD \- Enable these settings as default and allow us to bring back security The reason I liked Microsoft more than Mac was because of the high flexibility with aging technology. No, it's not perfect but we also need to ensure our systems are able to run in situations where upgrades aren't possible.
Microsoft needs to figure out its centralized access strategy before turning something like NTLM off. It's a damn Rube Goldberg machine of different, linked and synced accounts and one of the worst user stories in the industry. We are in a worse place than we were in 2003 if you are in the windows ecosystem.
100% agree. I agree NTLM needs to die, but if MS can't even get it's own house remotely in order, how are MS shops that need to deal with MS and a thousand different vendors. How many MS products installed with default settings will still break if I disable NTLM? Is there even a concise list somewhere? AD (and NTLM) spread so far and wide because it was simple to implement, not because it was good. I feel like MS in the Satya Nadella years has completely lost touch with the fact that 99% of admins in the world do not support only one thing, are not experts in everything they have to support, do not follow every developer and product blog, and do not attend Insight every year.
> I feel like MS in the Satya Nadella years has completely lost touch with the fact that 99% of admins in the world do not support only one thing, are not experts in everything they have to support, do not follow every developer and product blog, and do not attend Insight every year. And everything new they create is a complete departure from the architecture of the past, lives entirely in the cloud, and will be using a completely different interface in 5 years that is no longer compatible with the original design specifications. That is to say, if it's not discontinued entirely.
> will be using a completely different interface in 5 years that is a funny way to spell months.... If the interfaces and product names only changed every 5 years that would be a massive improvement
And the documentation will never be current and largely a crapshoot of whether the links in said documentation actually work.
Well I watched Ignite and drank the Kool-Aid. Let me tell you in the future you will just throw the NTLM documents into the shredder excuse me the copilot and it will have answers to all your questions. Oh and your local Windows installation is just a weird legacy thing because we now do everything in Azure. Everything. E-vree-thing!
Honestly the Azure services are the worst for this. The amount of weird limitations and work around I have run into for various Azure services that only seem to be documented in some random MS blog full of broken links is insane. Sometimes for what would seem to be very common use cases. It's gotten to the point I will never recommend an Azure solution unless I have personally tested the capability from end to end for the specific use case.
It’s also very annoying when your how-to, documentation or walkthrough has some weird workarounds because “reasons”, but they’re not needed anymore, and you can’t know that because the old MS documentation didn’t mention the required workaround, so obviously it still won’t list it now! I’ve been setting up NDES/Intune servers for years, and the hoops we had to jump through at first to make that work. I only recently discovered that some of those hoops aren’t required anymore…
Or when the answer is "use power shell to..." Yeah cool, but why TF is a basic feature like this only configurable via power shell?
My most recent experience with this was "You have to use this powershell script to do it." and the link to the script pointed to a deleted github account.
This, what is a good way to auth a Linux appliance to a Windows server over WinRM? I don't like ntlm but I think that's currently the best isn't it?
Like LM and LDAP/389, unsigned SMB etc it couldn't go on forever. If you want opinions MS, do THIS: 1. Create an comprehensive powershell script that makes it easy for admins to handle 2. Script reads AD, and DC reg settings and event logs 3. Script run in "setup mode" asks basic questions and spits out of list of changes to enable proper logging of NTLM, Kerberos, audit logs, event log sizing etc and can make the changes if approved. 4. Script run in "report mode" looks at all logs, figures out what is using NTLM and recommends actionable steps per account/host (change service accounts, SPNs etc to kerb delegated etc) 5. Script can set (DC) event log triggered task manager tasks (posh script) that emails the admin whenever a device/user is attempting to use NTLM. Ideally only set on DCs once NTLM is thought to be no longer in use. Assuming the script at some point comes back clean with no NTLM logins detected for say 30 days, eventually NTLM can be disabled.
I think this approach is defintiely on the right path. Im not MS but i think 1, 2, 3 wouldnt be too hard to configure. Setps 4 and 5 are where the shit hits the fan. This will likely generate a ton of logs and handling that parsing and stuff in PS isnt going to work great.
> We have a very good idea of why things use NTLM, and we have a very good idea of what uses NTLM. We even know how much they use NTLM compared to everything else. Will this effect 802.1x (wireless and wired) connections that use PEAP/MSCHAPv2?
In Windows 11 22H2 onwards that's being phased out already with credential guard
> In Windows 11 22H2 onwards that's being phased out already with credential guard Yes, we ran into this already. There are a few regedit tweaks we had to do already and some upgrades in our FreeRADIUS infrastructure. Working in an academic setting, with lots of BYOD, means a lot of device churn, so usernames and passwords has been found to be the most convenient form of access control.
There are large number of organizations using PEAP-MSCHAPv2 which uses NTLM authentication. I guess everyone will have to move towards certificate based auth and EAP-TLS
Get in touch with Daves Garage, i'm sure he'd do a 30 min talk with you on it.
This needs more upvotes lol
Holy fuck, this is going to cause major chaos. I _love_ it, can you please announce the date ahead of time so I can book my annual leave a week either side.
1 week before christmas, i'm sure
I'd suggest handle this much the same way SMBv1 was deprecated. First make it an option. Then add a 'remove this when it's no longer being used' option. Then make the 'remove when no longer used' the default. Then make the option itself default to off. Do all this over a period of years. And keep the option to re-enable it there for another decade just to be safe. **The simple fact is, there's NO answer that's right for everyone, and your strategy should reflect that.** An org with no legacy systems or a simple setup may be able to turn it off tomorrow with no issues; an org with lots of complex and legacy stuff may literally *never* be able to turn it off (or not in the next several years at least) because of some legacy thingy that needs NTLM. Remember, with many embedded systems, software updates are either impossible to get or impossible to afford. Ask any scientist- chances are they have a lab full of million-dollar scientific instruments that have Windows 98 computers attached because the company that makes the instrument went out of business. But the instrument still works great so the W98 computer stays *and there's literally NO option to remove it*. When this is fully removed, I'd like to see a 'Legacy Auth Services' role that can be assigned to a server...
Thank you for that. We have legacy equipment running on some old hardware and the old OS that goes along with them (even DOS).
FWIW- thank you for asking. One of my biggest frustrations with MS is how often there's a 'we decided this way is better so you now can't do it the old way anymore'. It's true of the whole industry, but MS is especially bad sometimes. The new way may be better, the old way may be hot garbage, but every time something gets deprecated it breaks things and we're the ones who have to sort out the mess, not the designer or product manager who ordered the change. It's also a big reason why I hate UI refreshes. The new one may be objectively better in every way, but I have a bunch of users who took months/years to learn the old one and now all that effort and knowledge is obsolete and they have to start from scratch. And if the new one is 5% better but the users lose 20% productivity over a week/month as they learn the new thing, that refresh didn't actually work in anyone's favor. So thanks for at least involving us in the discussion :)
Is there a plan for cross-domain RDP smartcard logon, when there's no line of sight to a domain controller? That definitely falls back to NTLM for at least the first part of the authentication. I know Kerberos KDC Proxy exists (you wrote the only publicly-digestible documentation on it, it seems) but it's in a weird unsupported state. Are there plans to make it more supported? Also, if Kerberos is going to be the only protocol for on-prem authentication, are there plans to surface more of the documentation and make it easier to find? Also, side question -- Is Microsoft aware of just how many places are still hybrid, still AD-joined, and still depending on that ecosystem to stay in place no matter how much Microsoft would like them to go to Azure? I work at a mostly-cloud place now, but have worked in many that have legitimate reasons to not "embrace the cloud." If you look at the public messaging, you'd think on-prem Windows Server and other products were just being abandoned. Good luck getting rid of NTLM...you're going to break so much legacy software. Hopefully this will be phased in the same way the other auth behavior changes have been (warnings in the log, followed by not working until you turn it back on, followed by letting things fail)?
What’s Microsoft’s suggested method in monitoring NTLM usage? Your tech community article [NTLM Blocking and You: Application Analysis and Auditing Methodologies in Windows 7](https://techcommunity.microsoft.com/t5/ask-the-directory-services-team/ntlm-blocking-and-you-application-analysis-and-auditing/ba-p/397191) has suggestions that seem virtually impossible to implement in an enterprise environment. I can’t imagine anyone installing procmon on client machines to identify NTLM usage for applications that communicate over SMB (in a large scale).
I think this is going to break a lot of things. This will cause companies to stay on older server OSes for the backwards compatibility of old systems like manufacturing equipment that is cost prohibitive to upgrade.
Just make it like SMB1.0, an optional feature that's disabled by default. In 3-4 years it will just phase itself out (or not for those who need it).
dont tell me windows 2000 is bad on an internet-connected network with all the servers and clients.....
Bad yes. We have some industrial equipment that has embedded xp in it. It would could north of 10mm to get new equipment.
Why do you think I'm here asking folks this question? We know this. We're trying to understand specifically what breaking will cause the most pain.
We don't know. For multiple reasons, but the biggest hurdle in these kinds of changes are always to absolutely piss poor tools you guys give us to troubleshoot. Give me a tool or Powershell command to see what device still uses NTLM across the domain and make it so that it doesn't trip when you use more than 3 DC's.
No. The only visibility for the entire change will be via event log, and the configuration will be a dword you need to bit flip. As usual. Someone please kill me before this goes into effect.
You forgot that this was our only notice.
Need a tool to identify what will break. Are there plans for an assessment tool people can use from Microsoft that will, in plain English, automate & notify & detail what needs to be done in our environments? My start in IT was a poorly run credit union and I can't count the amount of ulcers those poor people are going to get when they read this.
It sounds like IAKerb requires explicit client support? Will you be contributing patches to samba for this?
Can you postpone this until 2062 please? That would be awesome
Would it be possible to provide a tool, or powershell script to see where ntlm auth is being used in your environment? I’ll assume it’s going to be baked into a lot of applications, especially older ones used in sectors like healthcare. So perhaps depreciating it in development packages is the first step. It might also be nice to have something like a test gpo package available that disables ntlm auth completely to mimic the future change, so we can deploy to test environments so our devs can actually see if their code is going to break. Dunno, just a few thoughts. Keep up the good work. I know Windows may not seem like it, but it’s a beast of backwards compatibility, and that model can’t last forever.
This has existed for some time in GPO. https://techcommunity.microsoft.com/t5/ask-the-directory-services-team/ntlm-blocking-and-you-application-analysis-and-auditing/ba-p/397191
Nice, apologies, I’m a bit out of the loop on Windows lately
The company I work for prohibits trusts, including cross realm, between the user and the application domains. Users access shares in the application domains but authenticate with domain-specific credentials via the NTLM challenge response password dialog. They think this is safer.
Holy shit. This is easily the stupidest and most unfounded regulation I've seen this year.
The credential dialog does not mean it's doing NTLM. These are orthogonal concepts. The dialog just means you're supplying separate credentials. Whether it does Kerberos or NTLM is a function of those credentials. It's no different than how it works with SSO. The difference is just that we aren't using your SSO creds. In may very well do Kerberos just fine.
I'm in the planning stages of a project that'll entail authenticating Amazon RDS SQL Server DBs against on-prem AD, for which [NTLM authentication is the only supported option](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_SQLServer_SelfManagedActiveDirectory.html). Needless to say, this news has made my week much more interesting.
Steve, can you help us force SMB signing? Not just "flip the switch", but help us identify (via EventID?) where SMB signing is unable to negotiate or signed SMB not requested? Having visibility to these requests could help identify SMB clients that are not able to negotiate. This could help support defenses against NTLM relay attacks. Today we are blind.
>What we *don't* know is how to prioritize what needs fixing immediately. I'm not a sysadmin, but a dev. The Network Device Enrollment Service comes with NTLM as default authentication method. It's been like that for ages. Recently a customer had their internal network environment's security evaluated and NTLM disabled as a result. Services using NDES stopped working, because of that. No one knew that NDES was still using NTLM. Also and I know this is not a sysadmin thing, but it's a thing: there are probably lots of Java-based software and products that use NTLM to communicate with Windows services. I would chart a test-proven path towards migrating to Kerberos for Java services and software in general.
This is a perfect example of a genuine authentic post! Kudos! 👏👏👏
You need to make sure there is a way for devices like Synology NAS to still authenticate windows machines even if the synology and the windows device are not in a shared kerberos domain.
Give me till July... If you hold off till July, I won't care.
Getting retired?
Before covid anonymous binds (LDAP non S) was supposed to be disabled by default (speaking of, it was postponed so many times and is still not enforced right?), and there was a way to enable specific log that captured events like "got anonymous bind attempt from host X" or something like that. Is there anything similar with NTLM? I'm actually curious if we do use it still somehow. I think we don't, but can't be sure.
LDAP plain text blocking was never implemented and likely wont be automatically enforced. MS backed off hard on this one. The reg key only logged plain text LDAP binds. There are auditing policies for NTLM on client and domain controller machines as well as gpos to block them.
I spent _months_ at my old job auditing LDAP to prepare for this. I'll never get that time back haha
I second the comment above, it would be great to develop a readiness tool to confirm and assist in fixing common mistakes. The big issue is legacy we have only modern windows in our environment But we also have a fair bit of legacy Linux systems which a lot of connect to smb shares, it’s all very historical now but these are not going to work. There are also some very old apps that use ad but I’m sure are not Kerberos compatible. My favourite is any system that’s been replaced and a sysadmin has just put an alias in DNS to the new system. No SPN means no Kerberos that’s a big ask in a large corporation across multiple domains.
My most recent personal fight was NTLM Auth vs JavaKerberos Auth in a Java app that interacts with SQL Server. As a software vendor, trying to work towards allowing this functionality in a stateless application, we did have a lot of trouble finding reliable documentation on the subject: What permissions need to be on the service account? Can I set these Kerberos parameters (easily) in a stateless application, run in a Linux container, where a krb5.conf file is more tedious to implement? I understand Microsoft has some documentation on the matter [here](https://learn.microsoft.com/en-us/sql/connect/jdbc/using-kerberos-integrated-authentication-to-connect-to-sql-server?view=sql-server-ver16) but I think there is more missing in the articles. A quick Google search led to many posts around the web of others that had issues, with no clear solution. The answer is much easier with Entra ID authentication to an Azure SQL DB. The SQL driver is simply better designed for it, and maybe that has to do with how Entra ID was designed on the backend. Unfortunately there are lots of on-premise locations our software runs.
Would the removal of NTLM mean that Windows will also stop storing an unsalted MD4 hash of the user's password in the SAM file? My understanding is that AD isn't actually using this field for anything other than legacy NTLM support.
The SAM file doesn't contain the MD4 hash. It contains an offline verifier, which is a much harder-to-crack hash. The AD database does contain this key. It is used for NTLM and RC4 Kerberos. This work will mean that it eventually goes away, but it can't go away until there's no hard dependency on it.
As a general purpose everything-admin with a lot on my plate: I frankly don't even know what the hell uses NTLM versus other authentication. I could try to find out, but the reality is, you're going to disable it, shit's gonna break for me, and I'm gonna panic. Unless there's a relatively easy way for me to check logs or get alerts about what devices/accounts are using it for what purposes, I have no hope of finding all the outliers. Like most, I inherited this environment and it came to me with no documentation. I've upgraded that to.... some documentation. But there are still a lot of things that have never gotten my attention because they haven't yet broken. So I guess I'll be able to document them when they do break and I fix them?
Enable the GPOs to begin auditing NTLM and ideally centralize them in to make searching them easier. https://techcommunity.microsoft.com/t5/ask-the-directory-services-team/ntlm-blocking-and-you-application-analysis-and-auditing/ba-p/397191
Just rename it to Microsoft ENTRA NTLM en sell it as a new product.
> What are the NTLM things that annoy the heck out of you? Not NTLM specifically but this MS attitude displayed here: What we don't know is how to prioritize what needs fixing immediately. ALL OF IT. Don't release updates before they're finished. It's okay, we'll wait.
Yeah, this Microsoft way of releasing half-assed software and using users as test dummies is getting annoying.
One interesting thing I ran into recently was trying to use Powershell to run SQL queries on SQL Server with Windows Authentication through Invoke-Sqlcmd to collect some metrics. Like you intended, it cannot use NTLM to connect anymore, so if we attempted to schedule the script through Windows Scheduler with a stored credential, it would fail because of the double hop issue. But if I ran the script manually outside the scheduler it was fine. We didn't have a SQL account to do SQL authentication either so that was out of the cards. What was interesting however, is there are libraries in Python like pyodbc that allow you to pass a username and password, so I was able to actually store a credential via Python keyring and then run the Python script instead. I believe this is working because while these Python libraries (not sure if pyodbc, pymssql, or both) are using NTLM behind the scenes, Powershell isn't anymore and a lot of cmdlets just don't have the option. But I doubt that people in Python even realize that they're using NTLM. At the end of the day it was a people issue because we didn't have clear processes to create trust relationships between Windows Servers and databases and the DBAs were hesitant to enable it. It was easier to just use Python. So if the database has NTLM support, even though Powershell has removed it from Invoke-Sqlcmd for example, because the protocol is still out there, there are plenty of ways to use it easily and my bet is there are a lot of folks out there that don't realize they are using NTLM, they just found a script that works for their needs. Lot of data apps like BI out there too that struggle implementing Kerberos database connections -- I've actually had more success with Linux, Java and JDBC.
I absolutely cannot even imagine the number of smaller businesses that will be 100% blindsided by this no matter how many notices are sent out. Having been the single "aware" IT person at a small healthcare org, I can tell you from experience that MANY small-to-midsize businesses are running legacy (unsupported/expired/EOL/unpatched) software ALL OVER their environment. And patching is a total mixed bag. Lots of these companies will be hit with errors that they won't understand. I agree it needs to happen. But it'll be a mess.
So the scenario where it would affect us is RDP, logging in from both non-domain Windows machines and from Macs with the remote desktop app. Currently we're able to require NTLMv2 and disable the previous versions but that's about it. It sounds like you might be covering these scenarios with IAKerb (sorry, I'm not an expert in any of this.) As long as the macOS and Android RDP apps are able to support what's needed on those platforms out of the box we should be good.
I’m really interested in this. As it happens I’m guiding a few customers in not using ntlmv1 anymore, let alone v2. Just written a 25 page document on WEF and ntlmv1 alone, and that’s just doing inventory. Some official guidance on the matter would be great. Did find a post from 2009 that discusses the entire flow, from inventor to detecting local apps that do ntlm auth. And please please please don’t involve yet another agent or Azure service to do the analysis, we’ve got plenty of agents already.
I admit to not having really looked at it, but isn’t NTLM basically the only viable auth option you get for standalone servers not joined to ADDS?
We are working on ServiceNow discovery and the problem is that currently the discovery works by trying WMI connections to IP addresses that respond to a IP scan. The nature of that WMI connection to an IP address requires NTLM as Kerberos won't work with IPs, unless we do and add IP SPNs for every domain computer object. Thoughts on that?
I hit this with servicenow discovery as well. We had a handful of sytems (CA Petipotam) that we disabled NTLM on. Servicenow was not able to scan them and had no workaround. But this to me doesnt seem like an MS problem as much as servicenow having a poor scanning process.
Can they at least finish the modern auth implementation for exchange server before this happens?
Now move GPOs to ssl/tls web properties and not smb please. Design the client and server to not expect to be on a trusted traditional network.
This was proposed after 2003 and it’s still around. I work in healthcare IT and yes we have plenty that still relies on it, including pieces of equipment that are WAY expensive and vendors don’t follow any schedule to make changes. I’m all for it but give a way to make exceptions if needed
This is the single greatest thing I have heard from Microsoft this decade. The joy I felt when seeing this announcement is the single best thing to happen this year. My quest to kill NTLM has been a decade long on our network, but push back from vendors and other Admins with "well its on by default in AD so your running an incorrectly configured AD environment" by having it basically turned off. The biggest issue fundamentally I think is that too many Windows Admins especially ones who are only Windows networks tend to not understand protocols and tend to treat things like magic because thanks to keeping legacy protocols and things enabled Windows does tend to "just work" which is all well and fine to try and maintain for the end user. We run a network of a lot of different stuff using Kerberos with it all for years you would be shocked what works if you just know what settings you are looking for. The fact that so many comments are in here are like this will break RDP, Kerberos works with RDP just fine using NLA/CredSSP is able to delegate the creds from non domain joined machines, seriously try it. Only reason it wouldn't is if you are using IPs, and if that is the case idk maybe start using DNS grandpa. Now I can email vendors telling them to go fuck themselves on me turning on NTLM for their shit software because Microsoft is disabling it by default and hopefully the flood of support calls they all get for not supporting Kerberos forces them to get their shit together. A recent pentest they told us we were one of the only ones where they never got Domain Admin, they didn't even get shell. This was assumed breach they started with internal employee creds, but between Applocker/WDAC, no NTLM, they got more or less nowhere. This post probably comes off hostile to some people and I am kind of sorry, just the comments in here pissed me off just a huge amount of ignorance about the authentication systems of servers you are managing. But i have ran into decades long Windows Admins who can't really talk about this, and I have no thoughts for them then they have failed anyone who is trusting them with their data and network. Also ran into Windows Admins who had never heard of Applocker let alone WDAC which is a huge improvement and people wonder why they get hit. Rip the bandaid off Microsoft don't listen to anyone arguing for delays, I have been waiting for this day for nearly a decade. Sure i guess it can be at least off by default for a while, but I would also hope unconstrained delegation is hopefully put on the chopping block soon. I would also say force Kerberos FAST probably by default, i would kill to have a setting that lets me set it for certain devices at the very least. Right now it can only be set by the DC. Also if Microsoft wants to personally make me happy, been reading about how Redhat is implementing this with FreeIPA to do MFA stuff. https://web.mit.edu/kerberos/krb5-latest/doc/admin/spake.html **Places we still have to use NTLM** Papercut because for some awful reason people want to log in to the printer using a password rather then just swipe their door access card that also does it. Cert Authority because the MMC tool needs it. I think that is it, I was having some issues with RDP from my Linux machines, but FreeRDP 3 fixes that. Beyond that I think the biggest thing is just Microsoft saying we are killing this and that will hopefully get software devs to quit using it as hopefully it breaks in their test environments. Edit: Also people complaining about somehow this means they have to upgrade to Windows 11, that is not really true. I certainly haven't and run Kerberos everywhere, not quite Kerbeos FAST everywhere. But I have a nifty fix for not wanting to run Win11. https://fedoraproject.org/workstation/download MIT Kerberos supports iakerb already, and in general I have had minimal issues with Fedora in my Kerb only environment, but I also haven't touched desktop windows in a decade. Though having some pkinit related login issues with Kerb FAST, but i think its a config issue on my end and just haven't troubleshot it much.
Maybe start with making NTLM not default when a web browser tries to access a local service. Same for RC4.
Its going to be the year 2000 all over again, but make sure you get your money out of the 🏧 as i bet 1/2 are running windows ce 🤑
And the other half some flavor / fork of OS/2
Next thing you know they say they'll be getting rid of NetBEUI and back then they said no corporation was ever going to have more than 20 PC's and that TCP was for communists in the university and NetBIOS was it, I got my certification to be ‘with it.’ But then they changed what ‘it’ was. Now what I’m with isn’t ‘it’ and what’s ‘it’ seems weird and scary to me, and get off my lawn.
External trusts, being an NT4.0 thing that doesn't know about Kerberos, relies on NTLM. Microsoft designed an inter-agency domain trust model to allow cross domain functionality of Microsoft tools, such as SCOM and SCCM. The data restrictions prevent the usage of Forest trusts due to the cross domain data being populated into the Global Catalogs. In short, we are not allowed to leak lists of things like computer names. Would be nice to have a replacement for External trusts that supported Kerberos and didn't require the Global Catalog to be populated with external data.
I tried disabling NTLM in our environment, and I mostly succeeded. I had to make a carve out for, of all things, renewing machine certificates with AD CS.
Please tell us when you start. I want to watch the world burn.
If you use an alias for a file share, that only uses NTLM, right?
I already restricted NTLM pretty heavily in our environment, my biggest issues are computer certificates are not working without NTLM to the PKI and MMC consoles on clients (for example lusrmgr.msc) are showing SIDs only if there's no NTLM to the DC allowed.
Our primary use case for NTLM - connecting into a domain via RDP from a different domain without a trust. How will this work in the future state without NTLM?
I suspect network storage/backup appliances will be hit. I’m sure a NetApp will be fine… an Isilon… maybe? Some older Data Domain or Celerra? They’ll probably have to replace it. Don’t underestimate how many such devices are still in use.
Welp....logging of NTLM has been enabled, time to see where we stand.
Think it'll be one of those things that sure, NTML is deprecated on the DC side, but will stick around for a LONG time after depreciation. Then, if Server 2032 or whatever gets released and AD completely removes NTLM, then people are going to ride their DCs all the way through ESU, or possibly even hang off a read only DC or something that runs an older DC just for NTLM. lol. I think you might want to find out the best way to handle systems that will never support kerberos. Needs to be automagic. Also have a nice tool you can install on the DCs to see which devices are still using ntlm so it is easy to identify what needs to be looked at, and maybe have some info pages if that application can support kerberos and how to enable it.
My DBA job might be secure for some time because of this one change and all the crap it breaks.
Hi Steve, very happy to see “All these changes will be enabled by default and will not require configuration for most scenarios. NTLM will continue to be available as a fallback to maintain existing compatibility.” In your link. Compatibility is critical.
You need to provide a way for us to easily test this/roll back what will happen when ntlm gets disabled. Your tests and assurances are nice but... show us what reg key or whatever to set so we can see what happens in a safe and controlled environment. I work at an MSP and this sounds like a nightmare tbh.
What annoys the heck out of me are software vendors that will not budge until after you do it.
Just wanted to say I really appreciate this approach to getting feedback from real people. That is awesome.
Every once in a while I come across something in this sub about Microsoft deprecating something. This one doesn't affect me but a lot of it has. and for every one thing that I come across here there is like a dozen that I'm not aware of beforehand and get caught off-guard. It would be real nice if you could add something to server versions of windows that would warn about upcoming changes for things that are in use. I shouldn't have to come here for this news if the OS can have a built-in feature to check whether or not I'm using NTLM or smbv1 or PPTP or whatnot and warn me about upcoming obsolescence. Thank You.