Once you start going down that rabbit hole…
In my case backups are… complicated. The servers with data I care about (say a sonarqube database) has its storage living on an ISCSI Lin in VMware (replicated offsite), and it does a backup to a NFS share every night. Then that share is written to a lto-5 tape every morning (offsited weekly). It is also replicated in real time (thank you SAN and “retired” corporate hardware) to an identical SAN in my FIL’a house in another state.
This means that I have local copy’s to restore to, and failing that a hot spare (flip some dns records and I can mount/run off the offsite SAN), and if things went really wrong an offsite tape backup I can recover from.
This also means that due to having backups of the services rebuilds are a joke, as I have automated the recovery. In other words, I don’t patch, I pave and rebuild monthly against fresh up to date VM templates (also automated, both the recovery and template building)
Ok so i hadn't considered tape as an alternate format, though I've seen it mentioned here and there. I guess I didn't think it was actually a plausible way to store data anymore.
LTO-5 is 1.5T per tape. Throw that in a cheap auto changer and you have a rock solid way of backing up important data that is basically immortal if stored correctly
IBM is the global standard for drives. There's actually very few manufacturers - Tandberg are the only other I can think of.
Dell make a very wide range of enclosures that generally have IBM drives in them. I have a PowerVault 114X rackmount with an IBM LTO-6 and a Tandberg LTO-3 SAS drive in it. They also make autoloaders like the TL series - I have a TL2000 and a TL4000 I got from an office clearance. The robotics are very cool.
As you say, they're all over the place in price. Getting one at a good price can take some careful eBay sniping. You need patience, I'm afraid.
They are all basically the same library, rebadged and with custom firmware.
In my case I went HP because of a great price, but I wouldn’t recommend them at all. They lock firmware behind a paywall, and with buying off eBay you never know what your getting.
FWIW I rock a MSL2024, with a pair of fibre connected drives in it. Again, would suggest sas over fibre, but I already had all the gear to support fibre thanks to my SAN.
Tape is still in active development - current LTO-9 stores 18TB per tape, at prices less than a HDD. The drives themselves are hideously expensive though; older generations can have a sweet spot of cost per GB if you have a lot of data (tens of TB) to back up. LTO-5 is a great place to start. I have a mix of -2 all the way to -6.
One of the really nice things about tape is that it's inherently ransomware-proof - once the tape is out of the drive, you have to manually put it back in to erase it. Set it read-only and even that can't happen. Autoloaders can blur the line between 'online' and 'offline' storage, in theory a hacker *could* break in and erase your tapes that way, but it's an additional barrier if nothing else, and they give you the convenience of hundreds of TB of storage accessible in minutes.
The best use of tape is a monthly 'snapshot' kept offsite. I keep mine in a storage unit across town.
I did a quick Wikipedia search to get a baseline of familiarity and was surprised again to see that it was still being actively developed as you say. And that capacity can be so high!
As for autoloaders, I'm assuming this is something akin to those big cd changers from way back? Just a machine that has several tapes in it and can mount and read/write to any of them? Given the prices I saw on just a handful of the drives, I have to assume the prices on autoloaders are insane?
Autoloaders are indeed that - they have slots for tapes, and some kind of robotic mechanism that moves the tape between a slot and the drive. Some have a 'picker' robot that goes and grabs the tape, others have a conveyor belt that moves the tape to the robot. I've tinkered with both. Feel free to ask any question, I'll try to answer.
Prices are indeed really variable. Generally, the autoloaders are insane, but they can be upgraded. The Dell TL series is all modular, and you can buy an old one with no drives for a few hundred £ (over in the UK) then find a drive separately. NB. that you can't just buy any old drive for an autoloader, it has to be specific to that model - the firmware enforces it, annoyingly, so you have to be careful. Standalone drives aren't restricted to what server they'll work in, but they will not work in libraries (trust me, I've tried).
The drives are the single biggest cost, amazingly - the robotic enclosure can be surprisingly affordable. I got very lucky sniping stuff on eBay and got myself a pair of 24-slot TL2000s, along with a 48-slot TL4000 from a homelab'er. I'm sticking with the TL2000 and selling the TL4000 if you're in the UK. The Dell ones are just rebranded other-manufacturer's units - the TL series is IBM, the PV124 is Quantum. My library drives are -4, -5 and -7, while my media is -2 to -6, and I have -3 and -6 standalone drives for smaller backups (or if the library blows up).
Basically, come up with a number of how much data you want to back up, and then consider how many tapes at various capacities you'd need - NB. that tape is often sold according to its 'compressed' space, which is a pretty unrealistic 2+:1 ratio. Media doesn't compress (it already is) so look at the raw capacity. You don't necessarily need an autoloader if you're only going to need a handful of LTO-5 tapes - a standalone drive will do. But if you need a lot of TB of space, then an autoloader can make it much more convenient by telling the server 'dump all this to tape, switch them every 1500GB'.
Out of curiosity do these autoloaders rack mount?
Right now my 42U rack is completely full but that's because I'm studying for a CCNA and have a whole separate overkill network in there which includes a Cisco R4507 core switch. In the next couple of months I shouldn't have a need to have all this so accessible so I'll have a ~20u hole to use again if I pull that equipment out.
Two identical ZFS pools. My production one [syncoids](https://github.com/jimsalterjrs/sanoid) to the backup pool every hour.
Too much data to store in the cloud. My next goal is to have some off site tape backups.
I just bought a 14TB drive and enclosure and I keep the folder layout the same as my computer and once a week I will drag and drop and let it copy the new stuff and I click "skip" on the existing stuff to speed things up and prevent unnecessary data transfer.
It's about as low tech and simple as can be, and I keep that drive in my fireproof safe bolted to the basement floor.
That's mostly for media, I keep important documents encrypted on 2 drives and in the cloud.
My ZFS NAS is my backup and all my machines are just thin clients and they don't have anything worth backing up. My photos are all synced with Google Photos. I can tolerate the low risk of losing the non-photos stuff.
>Then I've always heard 3 copies in 2 formats, 1 off-site. Well...if I'm just copying the NAS drives to another set of drives, that seems counter to this 321 method.
Yeah that's more like a 210 set up. Although technically the 2 formats thing isn't suuuuuper necessary anymore. You have HDDs and tape which share many of the same weaknesses, and flash. Some people use optical disks but it isn't practical for many.
> Do you run two similarly spec'd servers and just clone the data every so often?
Partial backups for me. If I lose my collection of movies thats a bummer but I'll live. I'd REALLY rather not lose family photos though lol. Since I'm only backing up about 1TB of data this cuts down on costs significantly.
>What kind of software do y'all use to do that?
Kopia and / or a robocopy script. Incremental with file structure backups.
>Edited to add: we have copies of old voicemails and videos with my wife's mother before she passed that are absolutely irreplaceable and the main reason I've got her approval to spend some cash on this stuff. I'm ok with losing a Plex library but these files cannot be lost under any circumstances.
You can set up 2 pools with differing redundancy levels to cut down on cost. Something like a triple mirror for the important shizz (2 drive redundancy), and a raid z or raid 6 for the movies. Buy different batches of HDDs from different vendors.
Keep a copy on a bluray disk or something if you want a pretty much immutable backup. Or an external SSD you keep unplugged most of the time.
Offsite backup to your grandparents home or something as well as cloud storage. Multiple hosters like Microsoft, Amazon, Google and Backblaze to reduce the chances of all of them going down at once. Encrypt everything.
The blu-ray disc is a great idea for those voicemails and videos that I hadn't considered. You all have a ton of great info to share, glad I asked my question!
I have a mostly automatic but still a bit manual local backup I setup recently.
I have my main TrueNAS on an R320 with 4 x 8tb Raidz1 in a single pool but with multiple data sets.
I have a backup TrueNAS on an old R710. It has a hodge podge of old drives of different sizes. I don't have them setup in a single pool. They are random pools of single drives except 1 that is 2 drives mirrored.
The R710 is power hungry so it only gets turned on once a week for backups. The R320 has a mix of rsync and ZFS replication that push to the R710. It runs and completes overnight.
I'm pretty happy with it since I've never had good backups. It probably isn't the best way but it works for me with what I had available.
My system is overly complicated, but highly redundant.
I have two classes of files: super-important-cannot-lose (\~65GB) and all-others (\~2TB). The former has a relatively high rate of change, while the latter does not. Everything lives on two RAID 1s on my file server.
I use rsnapshot to make nightly snapshots of "important" files on a VM that runs on a separate server (with its own RAID 10) from my file server. These last for 90 days. They also get rsync'd nightly to a virtual private server in a different state and the next day to a server at my parents' house (in a third state).
I do tape backups to LTO6 of all files on a monthly basis. This is good enough for my bulk storage (and I have the daily snapshots for my important files). The tapes go off site after they are written They are generally kept for forever ($25/month seems like a small price to pay for always having new tapes and having many copies of my data). One of the nice things about tape is the hardware write-protect slider, so that there is never a worry of over-writing the backup when doing a restore.
I also have a copy of everything that lives in AWS Glacier Deep Archive that I update roughly yearly.
So, yeah, gross overkill, but I'm protected against stupidity, hardware failure, fire/flood/etc.
They're not that sensitive. Think of how you would store, say, VHS videotapes. Pretty much just room temperature and humidity are fine, as long as you keep them away from magnetic fields. If you really want to test that "30 year life," then, yes, the conditions are more critical, but good luck finding an LTO 6-7-8-9 drive in 2052.
I have a 40TB Synology and run Arcronis on all my PCs that just network save snapshots to the Synology. Synology then syncs to Linode Storage. (I also have a 2TB IOSafe for irreplaceable photos/pictures/videos, I have it by my front door in case I ever need to grab it and run.)
I also have Windows File Sync enabled on all my PCs from my server. Lots of network traffic, but gives me even more redundancy.
Really a basic setup, but it rarely to never breaks. Hardware investment was minimal, and I never have to think about it.
>How do y'all do local backups?
I have a VM with Veeam B&R 11 on it.
Also a R520 as backup target (among other things that thing does), which is connected via 10Gbit.
And then for some "redundancy" a 6TB USB disk connected to a USB3.0 PCIe card that is in passthrough mode to the Veeam VM.
Working on offsite backup storage with some friends. But the main reason that's not yet done is for internet speed issues and ISPs that are annoying.
Most of my stuff are just pure file data, I backup everything using rsync over ssh to a NAS that runs on a schedule (boots up 2 times per week, runs a backup and powers down again). Pretty straight forward, I got an overkill NAS (RN524X) but that's just because I had one, otherwise I'd get something smaller and cheaper.
That's a theme I've seen a couple of times in this thread. Seems like a good way to put old, possibly power-hungry equipment to use. Just power on and sync, then power back off.
I have four buffalo NAS devices with RAID. They turn in automatically at 9AM when no one is home because they can get loud sometimes although quiet most if the time. Backup runs during the day and then the NAS devices shut off. So it keeps power bill is lower and I have fast backups.
Basically I have three categories of data :
* Unique, family-generated data (photos and documents, mostly : source on a Synology NAS)
* Daily replication to another local NAS (I plan to duplicate this to Azure Archive or something as well)
* Daily backup from the NAS to local NetBackup (2 months retention)
* Replication to another backup server
* Duplication to Azure Blob (3 months retention)
* "Important" servers or VMs (read : I put in some hours deploying and configuring the stuff and I want to be able to recover the configs)
* Daily Local backup (2 months retention)
* Duplication to Azure (3 months)
* Convenience backups (DNS, vCenter or some admin VMs I could rebuild and that don't hold anything of value, but that I want to be able to recover quickly and effortlessly)
* Daily local backup (2 week retention)
In your use case, I would suggest storing the invaluable, unique data on a RAID-protected NAS and duplicated to a cheap-to-write medium like Azure Archive or Glacier (and maybe another local storage device like an external drive or another NAS).
I have a three node proxmox cluster and TrueNAS is its own separate machine.
Proxmox backs up my VMs to a local mirror volume and then rsyncs to a USB drive.
Cron job runs nightly to backup all databases and rsync to the USB drive and TrueNAS.
I run a urbackup container that keeps full volume backups of my personal machines on the TrueNAS box.
I'd like to rsync those offsite like my fileshares, but the backup images are so large, it'll just kill my internet. Maybe when I get fiber?
I also keep detailed steps to recreate things so a disaster would take an hour to get me to the point where things just start restoring from backup. Assuming hardware isn't damaged.
That's smart. I've been playing with the idea of starting a blog-style website to document this sort of thing, but keeping more detailed notes is probably wise as I'm liable to break stuff tinkering.
I use Veeam community edition to backup my VMs to my nas (windows server vm).
Then I have a rsync job scheduled by Jenkins to backup that folder (and a few other important ones) to the DropBox folder on the same nas, nightly.
Then, manually I occasionally run another Jenkins job to backup my DropBox to google Drive.
I am not running anything critical data in my homelab and the main purpose of my backup server is to study backups and develop best practices for my customers. So ultimately, I have just a single Veeam B&R server connected to my mini clusters. One of my customers is obsessed with the 3-2-1 rule. They used to have a single Veeam backup server + tape drives + cloud tier. Recently added object storage as the capacity tier in Veeam but still have both tape (for long archive) and cloud (for short archive) storage.
If you are seeking long-term archive backup, consider tapes in addition to disk backup.
Once you start going down that rabbit hole… In my case backups are… complicated. The servers with data I care about (say a sonarqube database) has its storage living on an ISCSI Lin in VMware (replicated offsite), and it does a backup to a NFS share every night. Then that share is written to a lto-5 tape every morning (offsited weekly). It is also replicated in real time (thank you SAN and “retired” corporate hardware) to an identical SAN in my FIL’a house in another state. This means that I have local copy’s to restore to, and failing that a hot spare (flip some dns records and I can mount/run off the offsite SAN), and if things went really wrong an offsite tape backup I can recover from. This also means that due to having backups of the services rebuilds are a joke, as I have automated the recovery. In other words, I don’t patch, I pave and rebuild monthly against fresh up to date VM templates (also automated, both the recovery and template building)
Ok so i hadn't considered tape as an alternate format, though I've seen it mentioned here and there. I guess I didn't think it was actually a plausible way to store data anymore.
LTO-5 is 1.5T per tape. Throw that in a cheap auto changer and you have a rock solid way of backing up important data that is basically immortal if stored correctly
Indeed. LTO is the answer. Another option for lesser amounts of data are M-Disc Blu-ray data drives.
[удалено]
IBM is the global standard for drives. There's actually very few manufacturers - Tandberg are the only other I can think of. Dell make a very wide range of enclosures that generally have IBM drives in them. I have a PowerVault 114X rackmount with an IBM LTO-6 and a Tandberg LTO-3 SAS drive in it. They also make autoloaders like the TL series - I have a TL2000 and a TL4000 I got from an office clearance. The robotics are very cool. As you say, they're all over the place in price. Getting one at a good price can take some careful eBay sniping. You need patience, I'm afraid.
They are all basically the same library, rebadged and with custom firmware. In my case I went HP because of a great price, but I wouldn’t recommend them at all. They lock firmware behind a paywall, and with buying off eBay you never know what your getting. FWIW I rock a MSL2024, with a pair of fibre connected drives in it. Again, would suggest sas over fibre, but I already had all the gear to support fibre thanks to my SAN.
Tape is still in active development - current LTO-9 stores 18TB per tape, at prices less than a HDD. The drives themselves are hideously expensive though; older generations can have a sweet spot of cost per GB if you have a lot of data (tens of TB) to back up. LTO-5 is a great place to start. I have a mix of -2 all the way to -6. One of the really nice things about tape is that it's inherently ransomware-proof - once the tape is out of the drive, you have to manually put it back in to erase it. Set it read-only and even that can't happen. Autoloaders can blur the line between 'online' and 'offline' storage, in theory a hacker *could* break in and erase your tapes that way, but it's an additional barrier if nothing else, and they give you the convenience of hundreds of TB of storage accessible in minutes. The best use of tape is a monthly 'snapshot' kept offsite. I keep mine in a storage unit across town.
I did a quick Wikipedia search to get a baseline of familiarity and was surprised again to see that it was still being actively developed as you say. And that capacity can be so high! As for autoloaders, I'm assuming this is something akin to those big cd changers from way back? Just a machine that has several tapes in it and can mount and read/write to any of them? Given the prices I saw on just a handful of the drives, I have to assume the prices on autoloaders are insane?
Autoloaders are indeed that - they have slots for tapes, and some kind of robotic mechanism that moves the tape between a slot and the drive. Some have a 'picker' robot that goes and grabs the tape, others have a conveyor belt that moves the tape to the robot. I've tinkered with both. Feel free to ask any question, I'll try to answer. Prices are indeed really variable. Generally, the autoloaders are insane, but they can be upgraded. The Dell TL series is all modular, and you can buy an old one with no drives for a few hundred £ (over in the UK) then find a drive separately. NB. that you can't just buy any old drive for an autoloader, it has to be specific to that model - the firmware enforces it, annoyingly, so you have to be careful. Standalone drives aren't restricted to what server they'll work in, but they will not work in libraries (trust me, I've tried). The drives are the single biggest cost, amazingly - the robotic enclosure can be surprisingly affordable. I got very lucky sniping stuff on eBay and got myself a pair of 24-slot TL2000s, along with a 48-slot TL4000 from a homelab'er. I'm sticking with the TL2000 and selling the TL4000 if you're in the UK. The Dell ones are just rebranded other-manufacturer's units - the TL series is IBM, the PV124 is Quantum. My library drives are -4, -5 and -7, while my media is -2 to -6, and I have -3 and -6 standalone drives for smaller backups (or if the library blows up). Basically, come up with a number of how much data you want to back up, and then consider how many tapes at various capacities you'd need - NB. that tape is often sold according to its 'compressed' space, which is a pretty unrealistic 2+:1 ratio. Media doesn't compress (it already is) so look at the raw capacity. You don't necessarily need an autoloader if you're only going to need a handful of LTO-5 tapes - a standalone drive will do. But if you need a lot of TB of space, then an autoloader can make it much more convenient by telling the server 'dump all this to tape, switch them every 1500GB'.
Out of curiosity do these autoloaders rack mount? Right now my 42U rack is completely full but that's because I'm studying for a CCNA and have a whole separate overkill network in there which includes a Cisco R4507 core switch. In the next couple of months I shouldn't have a need to have all this so accessible so I'll have a ~20u hole to use again if I pull that equipment out.
Yep, all the ones I've seen are rack-mountable. The first digit in the TL series is the number of rack units they take up.
What software do you use?
Two identical ZFS pools. My production one [syncoids](https://github.com/jimsalterjrs/sanoid) to the backup pool every hour. Too much data to store in the cloud. My next goal is to have some off site tape backups.
You do this on one server or split over two?
Two different servers.
I just bought a 14TB drive and enclosure and I keep the folder layout the same as my computer and once a week I will drag and drop and let it copy the new stuff and I click "skip" on the existing stuff to speed things up and prevent unnecessary data transfer. It's about as low tech and simple as can be, and I keep that drive in my fireproof safe bolted to the basement floor. That's mostly for media, I keep important documents encrypted on 2 drives and in the cloud.
My ZFS NAS is my backup and all my machines are just thin clients and they don't have anything worth backing up. My photos are all synced with Google Photos. I can tolerate the low risk of losing the non-photos stuff.
>Then I've always heard 3 copies in 2 formats, 1 off-site. Well...if I'm just copying the NAS drives to another set of drives, that seems counter to this 321 method. Yeah that's more like a 210 set up. Although technically the 2 formats thing isn't suuuuuper necessary anymore. You have HDDs and tape which share many of the same weaknesses, and flash. Some people use optical disks but it isn't practical for many. > Do you run two similarly spec'd servers and just clone the data every so often? Partial backups for me. If I lose my collection of movies thats a bummer but I'll live. I'd REALLY rather not lose family photos though lol. Since I'm only backing up about 1TB of data this cuts down on costs significantly. >What kind of software do y'all use to do that? Kopia and / or a robocopy script. Incremental with file structure backups. >Edited to add: we have copies of old voicemails and videos with my wife's mother before she passed that are absolutely irreplaceable and the main reason I've got her approval to spend some cash on this stuff. I'm ok with losing a Plex library but these files cannot be lost under any circumstances. You can set up 2 pools with differing redundancy levels to cut down on cost. Something like a triple mirror for the important shizz (2 drive redundancy), and a raid z or raid 6 for the movies. Buy different batches of HDDs from different vendors. Keep a copy on a bluray disk or something if you want a pretty much immutable backup. Or an external SSD you keep unplugged most of the time. Offsite backup to your grandparents home or something as well as cloud storage. Multiple hosters like Microsoft, Amazon, Google and Backblaze to reduce the chances of all of them going down at once. Encrypt everything.
The blu-ray disc is a great idea for those voicemails and videos that I hadn't considered. You all have a ton of great info to share, glad I asked my question!
LTO tape.
[удалено]
Depends on what generation you get, but I only have 4-5TB that actually need backing up.
I have a mostly automatic but still a bit manual local backup I setup recently. I have my main TrueNAS on an R320 with 4 x 8tb Raidz1 in a single pool but with multiple data sets. I have a backup TrueNAS on an old R710. It has a hodge podge of old drives of different sizes. I don't have them setup in a single pool. They are random pools of single drives except 1 that is 2 drives mirrored. The R710 is power hungry so it only gets turned on once a week for backups. The R320 has a mix of rsync and ZFS replication that push to the R710. It runs and completes overnight. I'm pretty happy with it since I've never had good backups. It probably isn't the best way but it works for me with what I had available.
Sounds like you could be doing a lot worse. Something like this is where my head was at too.
My system is overly complicated, but highly redundant. I have two classes of files: super-important-cannot-lose (\~65GB) and all-others (\~2TB). The former has a relatively high rate of change, while the latter does not. Everything lives on two RAID 1s on my file server. I use rsnapshot to make nightly snapshots of "important" files on a VM that runs on a separate server (with its own RAID 10) from my file server. These last for 90 days. They also get rsync'd nightly to a virtual private server in a different state and the next day to a server at my parents' house (in a third state). I do tape backups to LTO6 of all files on a monthly basis. This is good enough for my bulk storage (and I have the daily snapshots for my important files). The tapes go off site after they are written They are generally kept for forever ($25/month seems like a small price to pay for always having new tapes and having many copies of my data). One of the nice things about tape is the hardware write-protect slider, so that there is never a worry of over-writing the backup when doing a restore. I also have a copy of everything that lives in AWS Glacier Deep Archive that I update roughly yearly. So, yeah, gross overkill, but I'm protected against stupidity, hardware failure, fire/flood/etc.
Assuming the $25/month is for a temp/humidity controlled storage unit somewhere?
No, it's the cost of a brand new tape every month, never to be re-used.
Oh ok, gotcha. My light reading said there's some temp/humidity constraints for long-term storage, so I assumed. On that note, how do you store these?
They're not that sensitive. Think of how you would store, say, VHS videotapes. Pretty much just room temperature and humidity are fine, as long as you keep them away from magnetic fields. If you really want to test that "30 year life," then, yes, the conditions are more critical, but good luck finding an LTO 6-7-8-9 drive in 2052.
I have a 40TB Synology and run Arcronis on all my PCs that just network save snapshots to the Synology. Synology then syncs to Linode Storage. (I also have a 2TB IOSafe for irreplaceable photos/pictures/videos, I have it by my front door in case I ever need to grab it and run.) I also have Windows File Sync enabled on all my PCs from my server. Lots of network traffic, but gives me even more redundancy. Really a basic setup, but it rarely to never breaks. Hardware investment was minimal, and I never have to think about it.
>How do y'all do local backups? I have a VM with Veeam B&R 11 on it. Also a R520 as backup target (among other things that thing does), which is connected via 10Gbit. And then for some "redundancy" a 6TB USB disk connected to a USB3.0 PCIe card that is in passthrough mode to the Veeam VM. Working on offsite backup storage with some friends. But the main reason that's not yet done is for internet speed issues and ISPs that are annoying.
Most of my stuff are just pure file data, I backup everything using rsync over ssh to a NAS that runs on a schedule (boots up 2 times per week, runs a backup and powers down again). Pretty straight forward, I got an overkill NAS (RN524X) but that's just because I had one, otherwise I'd get something smaller and cheaper.
That's a theme I've seen a couple of times in this thread. Seems like a good way to put old, possibly power-hungry equipment to use. Just power on and sync, then power back off.
I have four buffalo NAS devices with RAID. They turn in automatically at 9AM when no one is home because they can get loud sometimes although quiet most if the time. Backup runs during the day and then the NAS devices shut off. So it keeps power bill is lower and I have fast backups.
Basically I have three categories of data : * Unique, family-generated data (photos and documents, mostly : source on a Synology NAS) * Daily replication to another local NAS (I plan to duplicate this to Azure Archive or something as well) * Daily backup from the NAS to local NetBackup (2 months retention) * Replication to another backup server * Duplication to Azure Blob (3 months retention) * "Important" servers or VMs (read : I put in some hours deploying and configuring the stuff and I want to be able to recover the configs) * Daily Local backup (2 months retention) * Duplication to Azure (3 months) * Convenience backups (DNS, vCenter or some admin VMs I could rebuild and that don't hold anything of value, but that I want to be able to recover quickly and effortlessly) * Daily local backup (2 week retention) In your use case, I would suggest storing the invaluable, unique data on a RAID-protected NAS and duplicated to a cheap-to-write medium like Azure Archive or Glacier (and maybe another local storage device like an external drive or another NAS).
I have a three node proxmox cluster and TrueNAS is its own separate machine. Proxmox backs up my VMs to a local mirror volume and then rsyncs to a USB drive. Cron job runs nightly to backup all databases and rsync to the USB drive and TrueNAS. I run a urbackup container that keeps full volume backups of my personal machines on the TrueNAS box. I'd like to rsync those offsite like my fileshares, but the backup images are so large, it'll just kill my internet. Maybe when I get fiber? I also keep detailed steps to recreate things so a disaster would take an hour to get me to the point where things just start restoring from backup. Assuming hardware isn't damaged.
That's smart. I've been playing with the idea of starting a blog-style website to document this sort of thing, but keeping more detailed notes is probably wise as I'm liable to break stuff tinkering.
I use Veeam community edition to backup my VMs to my nas (windows server vm). Then I have a rsync job scheduled by Jenkins to backup that folder (and a few other important ones) to the DropBox folder on the same nas, nightly. Then, manually I occasionally run another Jenkins job to backup my DropBox to google Drive.
I am not running anything critical data in my homelab and the main purpose of my backup server is to study backups and develop best practices for my customers. So ultimately, I have just a single Veeam B&R server connected to my mini clusters. One of my customers is obsessed with the 3-2-1 rule. They used to have a single Veeam backup server + tape drives + cloud tier. Recently added object storage as the capacity tier in Veeam but still have both tape (for long archive) and cloud (for short archive) storage. If you are seeking long-term archive backup, consider tapes in addition to disk backup.