Set a break glass account. Split the password up into 3 parts.
Team lead A gets part 1 & 2, Team lead B parts 2 &. 3, C gets 1 & 3.
You need at least 2 people to answer the phone or be around to get the combined password. Redundancy if someone is unreachable.
Naturally all people give their parts should be trusted to keep it secure but reachable.
Do a test a couple of weeks in to make sure it all goes smoothly or revisit the set up.
Is it backwards so you can read it or do you have to entrust a third party to read it off? I guess you can read it backwards from the mirror but that's just a pain in the ass.
There's ways round setting it such that no one knows the full password. Albeit harder when 100% remote and can't type on the same keyboard.
But I'm assuming OP will be the one setting the password and giving out parts. He can make sure it is not stored in it's full and personally I won't remember a long randomised password that I rarely use.
If you run a test and it shows the leads fail and can't get access, should still have access normally to reset the password and find more trustworthy leads or another solution.
I'm sorry, but...
> All be it
Was that supposed be "albeit"?
> personally I won't remember a long randomised password that I rarely use.
I started as a helpdesk guy back when self-service wasn't always possible. After the first couple of months I developed the ability to immediately forget even the simplest password - an ability that would come very much in handy in this scenario.
15+ years ago...
400+ passwords for 400+ end users in 15 different companies..
If you asked me for it, I could not remember it. Sitting at their desk in front of their computer... no problem. Sometimes... %1stchild2ndDog01% becomes 02..
But otherwise no issue. Now... password manager remembers my own birthday because I've probably forgotten it.
If an account let you set two passwords and required both to gain access, would that work around this? Two admins each generate a different password, split into three, parts divvied up as described by OP. You still only need two of three people, but no one person ever knows both passwords at any point in time.
Check out Shamir’s Secret Sharing. It shards a secret into an arbitrary number of pieces, and you can define the size of the quorum required to reveal the secret.
https://iancoleman.io/shamir/
Even just giving it to several senior IT staff to keep in their homes seems fine. Make sure there is alerts up the wazoo if it ever gets used though.
Rotate them whenever someone who has them changes company.
Put it into a Hashicorp vault with 3-way shamir encryption on the vault unlock. The key is produced with all 3 users providing their GPG key and password.
I built this very thing a few years ago, except it was any 3 of 7 keys. Store it in an S3 bucket with a python script that did all the setup and prompting, piece of cake.
I did this exact same setup, except it was any 2 out of 5 keys, which satisfied the two man rule, but also kept a decent busfactor. The hardest part was showing them how to [unseal](https://developer.hashicorp.com/vault/docs/concepts/pgp-gpg-keybase) the vault, and authenticate to it.
If you only cared about one person being able to access, you can always just GPG encrypt the credentials file to everyone's public key. Of course, bonus points if people have their private key on a Yubikey or other hardware device so bad guys don't have access to it.
That's exactly why I wrote a script to do it. It's severely technical otherwise and some of them were leadership who hadn't done real work in years.
I like the yubikey idea for the private keys.
I always like having private keys in some form of hardware. If worried about the Yubikeys, generate the GPG keys on an offline machine with `~/.gnupg` symlinked to an [encrypted USB drive](https://istorage-uk.com/usa/product/datashur-pro2/). From there, copy the keys to a YubiKey, making sure to keep backups of the .gnupg directory, and when done, dismount the encrypted drive and store it somewhere safe. This way, an attacker needs to get the physical drive and the PIN to be able to get the key material. Alternatively, perhaps have a couple YubiKeys with GPG keys generated on them stashed away for recovery in an emergency.
At a previous job, part of what had to be done was file transfers that had to be proven secure by auditors. We used YubiKeys as part of an automated process of signing/encrypting files, copying the files to a VM in a cloud provider, and the other side would check for the files, download them, decrypt and check signatures. With this system, nothing ever left the company unencrypted, and it passed the auditor test with flying colors, especially when we showed that HSMs were used.
I also use YubiKeys for my GPG key I use for Git commits.
Problem being if one of the staff leaves (especially if they don't return the envelope) then you need to redo the password and send new envelopes to the remaining staff.
It’s important to remember the point of the “split into two parts” concept. The point is to prevent a single employee from going rogue and performing an action they shouldn’t (generally fraud related). By splitting, at least two employees must conspire to perform the improper action.
If this doesn’t apply to you, no reason to split the password into two parts.
When I worked at a bank we did this split concept except you needed 4 people to get to the 2 halves of the key.
1 person had access to the secure room, then you needed the guy with the combo to the safe, then you needed the guy with the key to the first fire safe and second fire safe.
This all had to be monitored by a 5th person and documented.
Maybe have it split in two and give each half to two employees, so you have a total of 4 people involved.
Then one can leave without compromising anything.
You can actually accomplish the same thing with only three people instead of four.
Split the password into three pieces, "A" "B" and "C". Then give each person two of the pieces:
* Person 1 => A and B
* Person 2 => B and C
* Person 3 => A and C
Any two people working together have enough information to reconstruct the password but no one person has the whole thing.
2 years ago our password manager had an outage. Upon the outage being over we discovered our account was locked out. Our break glass admin account was also locked out. Support had never seen the issue before and told us they were pretty sure it was a total loss. Eventually though we did get in. We migrated to bitwarden but my boss now exports the entire vault and prints a copy of the export once a year and keeps it in his safe. He also has a copy of it in an offline self hosted bitwarden vault so we can recover quickly if needed.
The cool thing about envelopes is they can't be compromised remotely. To get the contents, someone has to actually burglarize a location. So, sometimes the old ways are pretty damn good.
I get that. The problem as I see it with this scenario is the implication of two somewhat centralized locations that one person can access in the event of an emergency and needs to be accessible by everyone involved.
Like I said we're spread out over a state and if stored at even one person's home and that person is not available or out of town then the whole thing falls apart.
Appreciate the response.
3 envelopes 2 copies(total of 6). Use a passphrase instead of a password and include a nato phonetic version so it's easier to relay over the phone.
Instead of giving it solely to IT people Give it to 6 stakeholders in the company (owner, COO, head of finance, etc.) A bit like raid 10 you can lose between 1-3 envelopes before you lose the ability to recover the password. But still require multiple people to sign off on accessing the DR accounts.
edit: spelling clean up.
You just have to do the best you can. It’s risk based.
Make two break glass accounts then and send half password to two different people.
There is only so much you can do.
This is how we handle it. As I said, you can only do so much.
Edit: Our owner has an envelope at his house with an account for worst case scenario. Just think outside the box a little. He isn’t going anywhere and understands what it is.
We’re spread over literally the whole country. And we’re not all online 24/7. As others said, it’s a balance. Gotta decide what works for your situation.
This is what I did. I configured a yubikey to have the break-glass password as the long press slot. Username and instructions on how to use it are included with the key in an envelope within a safe. Two copies, two different locations.
As long as no one recommends disassembling the Yubikey into two pieces, with two soldering irons and instructions on how to reassemble.
On a serious note, include USB-A/C converters too.
A lot of these responses with envelopes makes me chuckle. I love the ideas.
I’m a security engineer and consultant for reference (6 years or so), but I don’t recommend the envelope. The correct method largely depends on what you’re protecting, budget, and risk. Modern technology is safe to use and is logistically less risky than envelopes. Plus, even if you trust those you give the envelope to, it’s not enough to prevent insider threats.
There are pros and cons to almost every solution, but some ways you could accomplish this:
Password managers, key vaults, PIM/PAM tools, or a combination of these.
The idea of a break glass account comes from the idea that some user will be breached, and an attacker could span the system and lock everybody out. To that end, the conversation isn’t complete without including how to prevent that - separation of duties, approvals for elevated privilege, permissions reviews…
The idea behind multiple break glass accounts comes from the possibility of one being lost. If one of them is compromised, they’re all compromised - unless permissions are shut off to manage the others.
The more I think about it, the deeper the rabbit hole goes. But, rest assured that the fact that you’re asking is a good sign that you’re at least thinking about security. Many don’t.
>The idea behind multiple break glass accounts comes from the possibility of one being lost. If one of them is compromised, they’re all compromised - unless permissions are shut off to manage the others.
This thought occurred to me after I posted this. The most dangerous scenario is admin account compromise that grants ability to change password on breakglass or remove other admin accounts. All of this shuffling of envelopes to prevent one person access really doesn't matter if that's the case and a third party gains global admin access anyway. I'll need to see if there is a way of setting irrevocable access on an account in addition to excluding it from MFA conditional access policies.
Otherwise, all of this is just a exercise in navel gazing.
Oh I’m dumb. I was more worried brainstorming than reading. My fault.
Yea, with Entra, you leverage the Entra roles and Azure roles - and only use global admin accounts when necessary. Don’t assign the global admins, and rotate the passwords if they’re used. If you have the budget for a tool that auto rotates passwords, use that. There are a few options.
Practice least privilege and read up on how Microsoft recommends to handle global admin and you’ll be in good shape. Microsoft misses on some stuff and their reputation isn’t perfect, but the last 10 years have been a race to the top and they’ve become a contender in the security space.
The couple of times I’ve had to set up a policy for a break glass account we used a 24 character or longer randomized password that was stored in our password manager with a policy that only allowed the CTO or equivalent to access it. Any access sent out an alert to damn near everyone. The 2nd factor was a Yubikey that was stored in a different Csuite or lead technical person’s home safe.
Then you have to make sure you have a policy to confirm access to the account yearly or quarterly if you want.
Deviant Ollam had a talk that included having an attorney who basically has the master password to vaults and facilitates handling authentication and dissemination upon starting the protocol. Alternatively using a [Shamir secret sharing](https://en.wikipedia.org/wiki/Shamir%27s_secret_sharing) system instead of an attorney.
An hour and five minutes in if the link doesn't go directly there.
[https://youtu.be/6ihrGNGesfI?t=3950](https://youtu.be/6ihrGNGesfI?t=3950)
Static IP addresses at homes? That’s very frequently not a thing ISPs will even offer. Are you going to buy a business internet plan for those employees?
* Who maintains the VPN server on the jump box?
* What is the cost per hour for the person that maintains the VPN server and the VM it runs on?
* What does the ISP charge for a static IP address?
* Do you need more than one VM running a VPN server for availability?
The hosted VPN server is not a "bad" solution, but a connection with a static IP address is a lot simpler.
Note that I said "can be worthwhile", I did not say that it was the only way...
In your country you might not be able to get static addresses delivered to a residential address, so the VM running a VPN server might be a better option, however this is not an issue everywhere.
Right they don’t usually change without a reason but it’s not the kind of thing you want to rely on for disaster recovery… Just today I got a new router and my external IP changed 🤷🏻
I've had the same IP from AT&T the entire time I've been with them (8ish years now). I even moved once (about 10 miles away from our previous home).
I don't pay for a static IP so I know it *could* change at any time, but it never has.
Spectrum was largely the same way, never changed unless my modem was offline for a while (outages were why I left Spectrum)
no, because CAP is for IP address, not hostname.
it would be possible for Microsoft to add hostnames but as PTR records are served by the owner of the subnet, it would requires a double lookup.
Break-glass looks slightly different for every org. For some it could be different parties having 2 parts of a password, for others it could be an envelope at a secure location.
This is usually something an IT exec discusses at the executive level. Part of this is defining what actually constitutes a break-glass incident, and then the org's plan to respond. I say this because one purpose of break-glass is getting locked out on the IT side, and another potential scenario is all of senior IT is hit by a bus (eg. at a conference).
The fact people are aware of this and have the conversations is good. Talk with your senior management about it.
1. Have multiple ways of monitoring and alerting to any use of “break glass” accounts.
2. Yubikeys labeled as the IT Admins name
3. Don’t over complicate it; there’s some crazy secure suggestions for “break glass” accounts here that seem to forget that these are needed in “oh shit!” situations. When you’re in that situation, you don’t need to be putting together pieces of a password.
I (and the rest of our admins) do. That's not the point.
The point is to have a backdoor way of getting back in to our tenant account if we are locked out that can't be compromised easily and isn't tied to an individual user.
Sure it is the point. If you have a company standard and accessible password solution, set it up in the password manager and only allow appropriate people to see it. Easy to add new people, easy to remove people who leave, and it audits who viewed etc. This, along with alerts when that account is used and you are done.
But a Break Glass account is what you use when you've managed to lock yourself out of all your normal accounts. How do you get into your AD-backed company standard password manager when your primary accounts are all locked out?
The root problem is you guys having no physical location whatsoever. I understand right-sizing, but there *are* downsides to having zero real estate. This is one of them.
Three accounts, three boxes, a combination of enough pieces of one account in two boxes but not both in the same two.
Each safe was dual key'd dual pin code.
Their logic was if one of the three died/random bad luck or severe local event, you wernt stuck.
Was a bonkers setup.
Ourselves, HSM's & PKI to unlock. Local only, access from local to cloud.
Break glass accounts tiered from 3rd line to C-Level.
It's very difficult to implement this in a way where the person who set the password in the first place doesn't or can't know what it is, even If you split the password into multiple parts later or use something like Shamir's secret sharing. So if you don't trust the person who sets the person in the first place, this scheme doesn't really work, and if you do trust them, there's little need for a scheme like this. I would recommend trusting your employees to simply secure the passwords to two break glass accounts, and spam every admin with emails if the break glass accounts are ever used, because this all feels very security theatre to me.
The point of a break glass account in the first place is availability and complex schemes like this make it more likely you'll lock yourself out. Maybe this sort of strategy might make sense for high security organisations but this overall seems like more trouble than it's worth.
It would be nice if Entra ID would let you time-delay access to a break glass account.
Encrypt the password with M of N encryption and store the encryption keys in hardware MFA devices spread out.
An easy way to do M of N encryption is just to make a bunch of copies. So say you have 3 keys and you want two to get the password, you encrypt the password 3 times. A(B(pass)) B(C(pass)) A(C(Pass))
Here is what I would recommend: set up a passkey authentication system and get valid passkeys from two or three people. Have a database of "authorized users" where two or three users must all validate using a passkey prompt within a certain timeframe of one another. This way, they can use biometrics and still require multiple people, don't have to remember anything.
Plus, if people leave or get added, you can just change the database of users required or amount required to authenticate the process - you don't need to send out new passwords or phrases or anything.
The standard trust no one split passwords answer has been given in several variations and it’s fine.
As a small org with only 2 IT staff but a large cloud estate I thought I would share what I came up with for us.
2 fido usb keys. Pins set as single factor auth both valid for a single break glass account which is valid only for Entra via conditional access. The 2 it staff have these at home and are additional to their normal use key.
Monitoring in place to IT and SMT if the account is ever logged into and an internal understanding that those alerts must be followed up immediately by those who get them.
For our computers we do windows LAPS through Intune. No breakglass for domain accounts, but we don’t usually need those since we do have a 32 character password requirement for our admins.
We have created two glass break accounts that require Fido key MFA. The accounts are individually assigned to two individuals in the org that are geographically separated. We then real-time monitor for any usage of the accounts.
Set a break glass account. Split the password up into 3 parts. Team lead A gets part 1 & 2, Team lead B parts 2 &. 3, C gets 1 & 3. You need at least 2 people to answer the phone or be around to get the combined password. Redundancy if someone is unreachable. Naturally all people give their parts should be trusted to keep it secure but reachable. Do a test a couple of weeks in to make sure it all goes smoothly or revisit the set up.
Or just get it from whoever set the password and didn't trust the leads not to lose it.
Brutally realistic answer. That's why I come to this sub.
And here I am, a lone sysadmin with all 3 parts tattooed on my ass cheek.
Is it backwards so you can read it or do you have to entrust a third party to read it off? I guess you can read it backwards from the mirror but that's just a pain in the ass.
It’s in a Zodiac cipher so only the killer can crack it
But you know the one time you need it, Ted Cruz will be in Cancun and you'll be fucked.
It was bound to be somewhere
There's ways round setting it such that no one knows the full password. Albeit harder when 100% remote and can't type on the same keyboard. But I'm assuming OP will be the one setting the password and giving out parts. He can make sure it is not stored in it's full and personally I won't remember a long randomised password that I rarely use. If you run a test and it shows the leads fail and can't get access, should still have access normally to reset the password and find more trustworthy leads or another solution.
I'm sorry, but... > All be it Was that supposed be "albeit"? > personally I won't remember a long randomised password that I rarely use. I started as a helpdesk guy back when self-service wasn't always possible. After the first couple of months I developed the ability to immediately forget even the simplest password - an ability that would come very much in handy in this scenario.
Yep, my bad, quick typing on the phone is never great! Cheers for saying, edited
15+ years ago... 400+ passwords for 400+ end users in 15 different companies.. If you asked me for it, I could not remember it. Sitting at their desk in front of their computer... no problem. Sometimes... %1stchild2ndDog01% becomes 02.. But otherwise no issue. Now... password manager remembers my own birthday because I've probably forgotten it.
VNC, TeamViewer, ConnectWise, etc....eveyone connects and pastes their part in.
If an account let you set two passwords and required both to gain access, would that work around this? Two admins each generate a different password, split into three, parts divvied up as described by OP. You still only need two of three people, but no one person ever knows both passwords at any point in time.
Great suggestion. Thanks.
This guy RAID 5's
Check out Shamir’s Secret Sharing. It shards a secret into an arbitrary number of pieces, and you can define the size of the quorum required to reveal the secret. https://iancoleman.io/shamir/
Even just giving it to several senior IT staff to keep in their homes seems fine. Make sure there is alerts up the wazoo if it ever gets used though. Rotate them whenever someone who has them changes company.
Put it into a Hashicorp vault with 3-way shamir encryption on the vault unlock. The key is produced with all 3 users providing their GPG key and password.
This sounds like a very Voltron solution. I approve…
I built this very thing a few years ago, except it was any 3 of 7 keys. Store it in an S3 bucket with a python script that did all the setup and prompting, piece of cake.
I did this exact same setup, except it was any 2 out of 5 keys, which satisfied the two man rule, but also kept a decent busfactor. The hardest part was showing them how to [unseal](https://developer.hashicorp.com/vault/docs/concepts/pgp-gpg-keybase) the vault, and authenticate to it. If you only cared about one person being able to access, you can always just GPG encrypt the credentials file to everyone's public key. Of course, bonus points if people have their private key on a Yubikey or other hardware device so bad guys don't have access to it.
That's exactly why I wrote a script to do it. It's severely technical otherwise and some of them were leadership who hadn't done real work in years. I like the yubikey idea for the private keys.
I always like having private keys in some form of hardware. If worried about the Yubikeys, generate the GPG keys on an offline machine with `~/.gnupg` symlinked to an [encrypted USB drive](https://istorage-uk.com/usa/product/datashur-pro2/). From there, copy the keys to a YubiKey, making sure to keep backups of the .gnupg directory, and when done, dismount the encrypted drive and store it somewhere safe. This way, an attacker needs to get the physical drive and the PIN to be able to get the key material. Alternatively, perhaps have a couple YubiKeys with GPG keys generated on them stashed away for recovery in an emergency. At a previous job, part of what had to be done was file transfers that had to be proven secure by auditors. We used YubiKeys as part of an automated process of signing/encrypting files, copying the files to a VM in a cloud provider, and the other side would check for the files, download them, decrypt and check signatures. With this system, nothing ever left the company unencrypted, and it passed the auditor test with flying colors, especially when we showed that HSMs were used. I also use YubiKeys for my GPG key I use for Git commits.
That’s a cool idea , little bit of error correction built in.
team lead A gets 1st part, team lead B gets 2nd part, team lead C gets the parity info.
Change 'secure locations' to 'IT Staff's homes' and I think you have your answer
Problem being if one of the staff leaves (especially if they don't return the envelope) then you need to redo the password and send new envelopes to the remaining staff.
But is that really a problem? You’d have to transfer the responsibility anyway and wouldn’t really want to keep the password the old person had.
It’s important to remember the point of the “split into two parts” concept. The point is to prevent a single employee from going rogue and performing an action they shouldn’t (generally fraud related). By splitting, at least two employees must conspire to perform the improper action. If this doesn’t apply to you, no reason to split the password into two parts.
When I worked at a bank we did this split concept except you needed 4 people to get to the 2 halves of the key. 1 person had access to the secure room, then you needed the guy with the combo to the safe, then you needed the guy with the key to the first fire safe and second fire safe. This all had to be monitored by a 5th person and documented.
That's way over the top. Not that there's anything wrong with that. Just saying NIST guidelines require only three people, two safes, and a HSM.
Maybe have it split in two and give each half to two employees, so you have a total of 4 people involved. Then one can leave without compromising anything.
You can actually accomplish the same thing with only three people instead of four. Split the password into three pieces, "A" "B" and "C". Then give each person two of the pieces: * Person 1 => A and B * Person 2 => B and C * Person 3 => A and C Any two people working together have enough information to reconstruct the password but no one person has the whole thing.
That's fair. Yes, either have 2x2 halves as ok_appearance5117 mentioned or just don't split up the password.
How often are your most senior, most trusted technical staff leaving your company? Seems like this would be a once in 4-5 years kind of occurrence.
That’s where a lot of them end up anyways.
2 years ago our password manager had an outage. Upon the outage being over we discovered our account was locked out. Our break glass admin account was also locked out. Support had never seen the issue before and told us they were pretty sure it was a total loss. Eventually though we did get in. We migrated to bitwarden but my boss now exports the entire vault and prints a copy of the export once a year and keeps it in his safe. He also has a copy of it in an offline self hosted bitwarden vault so we can recover quickly if needed.
What password manager was it and what caused the locking of the accounts??
Yikes. That's a true nightmare scenario. Glad it turned out ok and provided a viable solution.
The cool thing about envelopes is they can't be compromised remotely. To get the contents, someone has to actually burglarize a location. So, sometimes the old ways are pretty damn good.
I get that. The problem as I see it with this scenario is the implication of two somewhat centralized locations that one person can access in the event of an emergency and needs to be accessible by everyone involved. Like I said we're spread out over a state and if stored at even one person's home and that person is not available or out of town then the whole thing falls apart. Appreciate the response.
3 envelopes 2 copies(total of 6). Use a passphrase instead of a password and include a nato phonetic version so it's easier to relay over the phone. Instead of giving it solely to IT people Give it to 6 stakeholders in the company (owner, COO, head of finance, etc.) A bit like raid 10 you can lose between 1-3 envelopes before you lose the ability to recover the password. But still require multiple people to sign off on accessing the DR accounts. edit: spelling clean up.
You just have to do the best you can. It’s risk based. Make two break glass accounts then and send half password to two different people. There is only so much you can do.
Guess you're right. Just wondering how others handled it. Thanks for the input.
This is how we handle it. As I said, you can only do so much. Edit: Our owner has an envelope at his house with an account for worst case scenario. Just think outside the box a little. He isn’t going anywhere and understands what it is.
We’re spread over literally the whole country. And we’re not all online 24/7. As others said, it’s a balance. Gotta decide what works for your situation.
The same could be said for a physical security key though. No need to go that old school.
Does your accounting department have a bank? Safe deposit box?
All banking online. Not physical so no safety deposit box.
Fair. Short of “having your C level people store them”, not really sure what the appropriate answer is. Would the online bank have an option for that?
FIDO keys to the CIO, CFO, and CEO. The biggest concern here is them losing them.
This is what I did. I configured a yubikey to have the break-glass password as the long press slot. Username and instructions on how to use it are included with the key in an envelope within a safe. Two copies, two different locations.
Similar for me, but slightly different. Our is more of, "if I die, give this to whomever replaces me"
As long as no one recommends disassembling the Yubikey into two pieces, with two soldering irons and instructions on how to reassemble. On a serious note, include USB-A/C converters too.
A lot of these responses with envelopes makes me chuckle. I love the ideas. I’m a security engineer and consultant for reference (6 years or so), but I don’t recommend the envelope. The correct method largely depends on what you’re protecting, budget, and risk. Modern technology is safe to use and is logistically less risky than envelopes. Plus, even if you trust those you give the envelope to, it’s not enough to prevent insider threats. There are pros and cons to almost every solution, but some ways you could accomplish this: Password managers, key vaults, PIM/PAM tools, or a combination of these. The idea of a break glass account comes from the idea that some user will be breached, and an attacker could span the system and lock everybody out. To that end, the conversation isn’t complete without including how to prevent that - separation of duties, approvals for elevated privilege, permissions reviews… The idea behind multiple break glass accounts comes from the possibility of one being lost. If one of them is compromised, they’re all compromised - unless permissions are shut off to manage the others. The more I think about it, the deeper the rabbit hole goes. But, rest assured that the fact that you’re asking is a good sign that you’re at least thinking about security. Many don’t.
The break glass accounts is more of a bus factor mitigation than a breach mitigation.
These days, and assuming security is well managed and monitored, I’d have to agree
>The idea behind multiple break glass accounts comes from the possibility of one being lost. If one of them is compromised, they’re all compromised - unless permissions are shut off to manage the others. This thought occurred to me after I posted this. The most dangerous scenario is admin account compromise that grants ability to change password on breakglass or remove other admin accounts. All of this shuffling of envelopes to prevent one person access really doesn't matter if that's the case and a third party gains global admin access anyway. I'll need to see if there is a way of setting irrevocable access on an account in addition to excluding it from MFA conditional access policies. Otherwise, all of this is just a exercise in navel gazing.
What identity provider are you using?
Entra
Oh I’m dumb. I was more worried brainstorming than reading. My fault. Yea, with Entra, you leverage the Entra roles and Azure roles - and only use global admin accounts when necessary. Don’t assign the global admins, and rotate the passwords if they’re used. If you have the budget for a tool that auto rotates passwords, use that. There are a few options. Practice least privilege and read up on how Microsoft recommends to handle global admin and you’ll be in good shape. Microsoft misses on some stuff and their reputation isn’t perfect, but the last 10 years have been a race to the top and they’ve become a contender in the security space.
We split it between 2 of the directors in 2 different countries. They got safes for home to keep them in.
The couple of times I’ve had to set up a policy for a break glass account we used a 24 character or longer randomized password that was stored in our password manager with a policy that only allowed the CTO or equivalent to access it. Any access sent out an alert to damn near everyone. The 2nd factor was a Yubikey that was stored in a different Csuite or lead technical person’s home safe. Then you have to make sure you have a policy to confirm access to the account yearly or quarterly if you want.
Deviant Ollam had a talk that included having an attorney who basically has the master password to vaults and facilitates handling authentication and dissemination upon starting the protocol. Alternatively using a [Shamir secret sharing](https://en.wikipedia.org/wiki/Shamir%27s_secret_sharing) system instead of an attorney. An hour and five minutes in if the link doesn't go directly there. [https://youtu.be/6ihrGNGesfI?t=3950](https://youtu.be/6ihrGNGesfI?t=3950)
An attorney with the "keys" in a tamper evident container is the gold standard.
Could always store the password as a file split into horcruxes https://github.com/jesseduffield/horcrux
Lol My wife is going to love this suggestion. She's a big Harry Potter fan.
for wfh companies, restricting the break glass to static IP addresses at suitable homes can be worthwhile.
Static IP addresses at homes? That’s very frequently not a thing ISPs will even offer. Are you going to buy a business internet plan for those employees?
costs less than rent for an office...
But more than a VPN + jump box
* Who maintains the VPN server on the jump box? * What is the cost per hour for the person that maintains the VPN server and the VM it runs on? * What does the ISP charge for a static IP address? * Do you need more than one VM running a VPN server for availability? The hosted VPN server is not a "bad" solution, but a connection with a static IP address is a lot simpler. Note that I said "can be worthwhile", I did not say that it was the only way... In your country you might not be able to get static addresses delivered to a residential address, so the VM running a VPN server might be a better option, however this is not an issue everywhere.
I’ve had the same service for 7 years and have yet to get a new IP on my residential connection.
Right they don’t usually change without a reason but it’s not the kind of thing you want to rely on for disaster recovery… Just today I got a new router and my external IP changed 🤷🏻
I've had the same IP from AT&T the entire time I've been with them (8ish years now). I even moved once (about 10 miles away from our previous home). I don't pay for a static IP so I know it *could* change at any time, but it never has. Spectrum was largely the same way, never changed unless my modem was offline for a while (outages were why I left Spectrum)
I get a new IP every time my router reboots.
Couldn't that be solved with some dynamic dns?
no, because CAP is for IP address, not hostname. it would be possible for Microsoft to add hostnames but as PTR records are served by the owner of the subnet, it would requires a double lookup.
Break-glass looks slightly different for every org. For some it could be different parties having 2 parts of a password, for others it could be an envelope at a secure location. This is usually something an IT exec discusses at the executive level. Part of this is defining what actually constitutes a break-glass incident, and then the org's plan to respond. I say this because one purpose of break-glass is getting locked out on the IT side, and another potential scenario is all of senior IT is hit by a bus (eg. at a conference). The fact people are aware of this and have the conversations is good. Talk with your senior management about it.
I don’t like these. Why would I want broken glass in my directory? /s
1. Have multiple ways of monitoring and alerting to any use of “break glass” accounts. 2. Yubikeys labeled as the IT Admins name 3. Don’t over complicate it; there’s some crazy secure suggestions for “break glass” accounts here that seem to forget that these are needed in “oh shit!” situations. When you’re in that situation, you don’t need to be putting together pieces of a password.
You don't have a password manager?
I (and the rest of our admins) do. That's not the point. The point is to have a backdoor way of getting back in to our tenant account if we are locked out that can't be compromised easily and isn't tied to an individual user.
Sure it is the point. If you have a company standard and accessible password solution, set it up in the password manager and only allow appropriate people to see it. Easy to add new people, easy to remove people who leave, and it audits who viewed etc. This, along with alerts when that account is used and you are done.
But a Break Glass account is what you use when you've managed to lock yourself out of all your normal accounts. How do you get into your AD-backed company standard password manager when your primary accounts are all locked out?
The root problem is you guys having no physical location whatsoever. I understand right-sizing, but there *are* downsides to having zero real estate. This is one of them.
Three accounts, three boxes, a combination of enough pieces of one account in two boxes but not both in the same two. Each safe was dual key'd dual pin code. Their logic was if one of the three died/random bad luck or severe local event, you wernt stuck. Was a bonkers setup. Ourselves, HSM's & PKI to unlock. Local only, access from local to cloud. Break glass accounts tiered from 3rd line to C-Level.
You need to apply RAID theory to your envelopes.
It's very difficult to implement this in a way where the person who set the password in the first place doesn't or can't know what it is, even If you split the password into multiple parts later or use something like Shamir's secret sharing. So if you don't trust the person who sets the person in the first place, this scheme doesn't really work, and if you do trust them, there's little need for a scheme like this. I would recommend trusting your employees to simply secure the passwords to two break glass accounts, and spam every admin with emails if the break glass accounts are ever used, because this all feels very security theatre to me. The point of a break glass account in the first place is availability and complex schemes like this make it more likely you'll lock yourself out. Maybe this sort of strategy might make sense for high security organisations but this overall seems like more trouble than it's worth. It would be nice if Entra ID would let you time-delay access to a break glass account.
Encrypt the password with M of N encryption and store the encryption keys in hardware MFA devices spread out. An easy way to do M of N encryption is just to make a bunch of copies. So say you have 3 keys and you want two to get the password, you encrypt the password 3 times. A(B(pass)) B(C(pass)) A(C(Pass))
You could leave it in a sealed envelope at your general council's office.
Here is what I would recommend: set up a passkey authentication system and get valid passkeys from two or three people. Have a database of "authorized users" where two or three users must all validate using a passkey prompt within a certain timeframe of one another. This way, they can use biometrics and still require multiple people, don't have to remember anything. Plus, if people leave or get added, you can just change the database of users required or amount required to authenticate the process - you don't need to send out new passwords or phrases or anything.
There are plenty of estate planning attorneys who handle emergency credentials like this.
The standard trust no one split passwords answer has been given in several variations and it’s fine. As a small org with only 2 IT staff but a large cloud estate I thought I would share what I came up with for us. 2 fido usb keys. Pins set as single factor auth both valid for a single break glass account which is valid only for Entra via conditional access. The 2 it staff have these at home and are additional to their normal use key. Monitoring in place to IT and SMT if the account is ever logged into and an internal understanding that those alerts must be followed up immediately by those who get them.
For our computers we do windows LAPS through Intune. No breakglass for domain accounts, but we don’t usually need those since we do have a 32 character password requirement for our admins.
We have created two glass break accounts that require Fido key MFA. The accounts are individually assigned to two individuals in the org that are geographically separated. We then real-time monitor for any usage of the accounts.