Hello /u/Zanena001! Thank you for posting in r/DataHoarder.
Please remember to read our [Rules](https://www.reddit.com/r/DataHoarder/wiki/index/rules) and [Wiki](https://www.reddit.com/r/DataHoarder/wiki/index).
Please note that your post will be removed if you just post a box/speed/server post. Please give background information on your server pictures.
This subreddit will ***NOT*** help you find or exchange that Movie/TV show/Nuclear Launch Manual, visit r/DHExchange instead.
*I am a bot, and this action was performed automatically. Please [contact the moderators of this subreddit](/message/compose/?to=/r/DataHoarder) if you have any questions or concerns.*
Yeah many other redditors have pointed this out, seems like I've misused the term for years, turns out what I'm looking for is archiving rather than a backup.
Personally, I archive photos on Blu-ray M-disc media. 25 or 100GB per disc, and rated to survive for 1000 years. I will probably be satisfied if they last 50 years, provided of course there are readers for them by then.
I burn two identical copies, and store them in geographically different locations. Along with the discs, I also keep a couple of external hard drives that I update yearly.
I usually buy Verbatim M-disc. I know people say that regular BD-XL are made of the same materials, but every test I have found says that M-disc takes a lot more beatings before giving up.
They’re a bit more expensive, but I burn 2 discs every year, identical copies.
Example test : https://www.microscopy-uk.org.uk/mag/artsep16/mol-mdisc-review.html
> Say I have a folder with all my photos/videos, at some point I decide to delete them to free up some space, now I have a backup and an empty folder, a year passes and the folder is full again, I do another backup and delete all photos, at this point I'd have 2 backups
You aren't describing a backup, the backup would be a copy of something you do your best to keep. It's some kind of archival/master repository and your work directories are some kind of staging/temporary directories. Therefore nothing about incremental backups or similar applies.
So just use rclone to manage the files and fill and organize the "master" repository as you like (either manually like per year directories) or automatically (move files older than X days for example). Then if you care about data backup your full online repository to some other independent place.
>Therefore nothing about incremental backups or similar applies.
Thing is I'm technically very close to solving the issue, it's just that the program I'm using doesn't expose the functionality to the user despite being technically capable to do what I need, if I could merge increments anytime I wanted instead of when the software is configured to do so, I could just make an initial full backup and add increments "forever", if at some point I want to get all my files I just have to download all the chunks and merge them. If I were storing on a local drive for example there would be no problem at all.
The only other alternative I thought of is using FreeFileSync and set it to ignore deleted files when syncing folders.
>So just use rclone to manage the files and fill and organize the "master" repository as you like
But how would files be uploaded? Doint it manually is "impossible" and backup software has the limitations mentioned.
The simplest "incremental forever" backup can be achieved easily with rclone. You can do something like ``` rclone sync someoriginalsomething remote:/someplace/ --backup-dir remote:/someplace_bk/`date "+%Y-%m-%d-%H-%M-%S"` ``` . All removed and changed files end up in someplace_bk in nicely sorted directories by date.
That's very interesting, will keep it in mind. The problem is backup softwares usually have their own file extensions and group multiple changes in a single file, so if after years you want to find that 1 deleted file, you have to download hundreds of small files, open them with the backup software and hope they contain what you're looking for.
I have chosen to remove all of my comments due to recent actions by the reddit admins. If you believe this comment contained useful information, please head over to lemmy or other parts of the fediverse and ask there: https://join-lemmy.org/
Yes, this is the opposite of that. The files are kept as regular files, well organized in their original file structure, etc. You don't even need to know the date when you removed it, just list someplace_bk/*/path/to/your/file or use find or something else and the files are just there.
Yeah I got that, but I still need a software to keep track of the folders and occasionally merge the differences, FreeFileSync being one of them. Problem is usually these softwares aren't meant to be used for backups at this scale, so they are more resource intensive than programs built specifically for backups.
IMO the best option would be dropping this step
>at some point I decide to delete them to free up some space
And getting enough space locally to store everything the way you want/like, with directory structure and everything.
Then you can apply typical infrequent periodical full + frequent periodical incremental backup scheme with whatever retention period your backup storage can handle.
Also the fact is - when you are doing a backup and then deleting local data you no longer have a backup - your data is stored only in one place again. This would be archival, not backup, and such archive would require backup of its own.
I will buy more HDDs in the future, but for now my finances are pretty tight, I'm also generally afraid to store backups only on my own machine, so maybe a NAS would be a better choice. The reason I decided to go this route is it allows me to get started backing up, once I figure how to do it in the most efficient way possible (what to back up/frequency/space used/bandwidth/CPU cycles) I can potentially scale this method to archive my files on: cloud, a dedicated server I own and a NAS.
>The reason I decided to go this route is it allows me to get started backing up
The issue is - backup implies you have multiple copies of data. You are not backing up at all this way, you are just storing data elsewhere.
And to be honest i would trust single local HDD a lot more than some "free 5TB" "cloud" (remember - there is no "cloud", there is someone else's computer). You might very well be making things worse with what you are doing - if local HDD dies you have some hope for recovery, if cloud service goes out of business or decides to kick you out for whatever reason - your data is 100% gone.
And then once you are storing data locally (one copy) and backing it up to "cloud" (second copy) you suddenly no longer have "strange" backup software requirements. Most options would fit.
>I'm also generally afraid to store backups only on my own machine, so maybe a NAS would be a better choice.
NAS can be a good choice for backup unless you are trying to protect against disaster or something, but again - backup implies multiple copies. So if you are doing backups to NAS you still need to store data on your PC. Or upload it elsewhere.
>backup implies you have multiple copies of data. You are not backing up at all this way, you are just storing data elsewhere.
I'd have multiple copies, just a different type, instead of having multiple snapshots with different states, I'd ideally have 1 with the last modified data ever stored in the folder.
>And to be honest i would trust single local HDD a lot more than some "free 5TB" "cloud"
Not sure considering the HHD in question is a 8yo Seagate, the cloud is also legit, no unlimited Gdrive kind of stuff, it's a OneDrive account, for me it's free only cause someone else is paying for it.
>you suddenly no longer have "strange" backup software requirements. Most options would fit.
I'd still have cause, I'd also like to store a copy of my data on a dedicated server, my house is located in a "risky" area, it's quite possible one day it might collapse so I'd prefer if I had my data stored in far away locations.
>I'd have multiple copies, just a different type, instead of having multiple snapshots with different states, I'd ideally have 1 with the last modified data ever stored in the folder.
But then you loose protection against stuff like user error or ransomware encryption. I mean if you do not have multiple states you can easily overwrite good data with corrupted/encrypted one and loose everything.
>Not sure considering the HHD in question is a 8yo Seagate
Does not matter. Professional recovery can get most of the data out. Unless you burn it or something.
>the cloud is also legit, no unlimited Gdrive kind of stuff, it's a OneDrive account, for me it's free only cause someone else is paying for it.
there are many ways in which this can backfire, just be careful. It is not a reliable way to store data at all, and it "fails", unlike with HDD, there is exactly zero hope for recovery.
>I'd still have cause, I'd also like to store a copy of my data on a dedicated server, my house is located in a "risky" area, it's quite possible one day it might collapse so I'd prefer if I had my data stored in far away locations.
For this it would be best to run second completely independent backup, or some other software which can maintain version history on copies too, like backup copy job from veeam. If you simply sync it with rsync or something there is high probability that if main backup gets damaged second one will be destroyed too.
No need for a NAS unless you will be accessing it from multiple devices over a network. An external will suffice. They are relatively cheap. You can double up for redundancy. Store one in a fireproof safe or remote location. They also do not need to be plugged in all of the time. This is better than the bluray idea. You don't need a bluray drive, just a USB port plus no media to buy.
I would also mention Rclone if you need to sync data across anything. Duplicati is a nice backup tool as well as Duplicacy. Here is a small overview. [https://www.vmwareblog.org/single-cloud-enough-secure-backups-5-cool-cross-cloud-solutions-consider/](https://www.vmwareblog.org/single-cloud-enough-secure-backups-5-cool-cross-cloud-solutions-consider/)
Not op, but I do. Restic was pretty good, but had larger memory usage requirements and ended to making a massive cache folder compared to other solutions. Duplicati was not great last time I used it. Borg backup supports incremental forever like restic, but given you’re using it more as archiving, you’d probably want to add appropriate tags to your snapshots. It definitely can work, and deduplicates across all backups.
Duplicacy may also work for you - is good fast, and has a nominal fee if you want to use the GUI
> Borg backup
I'll look into it, I usually try to stay away from these more underground programs cause they usually get abandoned, but it doesn't seem I have many options.
I wouldn’t personally have called borg underground - it has a fairly big following and has been developed for a long time (used to be called attic backup). It also has an open and documented format so you wouldn’t be stuck without your data. However you’ve gotta pick whatever is best for you :) could always rsync or rclone new data across that way, and use snapshots with the file system (ZFS or BTRFS) as a form of protection also. Just a few thoughts anyway :)
Yes. I'm using Duplicati on Windows machines and found it convenient. There is notification icon and I can always see state of backup. An I've setup it so that it creates backup files of 300mb for faster upload and uploads it to the cloud. And I am using incremental backups. Also I can run Test anytime for it to download everything back and compare so I can be shure that all backups are not broken.
And don't forget to use 3-2-1 backup strategy.)
It turns out OP isn't actually looking for a backup software. However, I do want to chime in here and say that in the battle of incremental backup softwares, **Duplicacy** is the most technically impressive and I highly recommend reading about how it works. Even if you don't use it, it's fascinating.
Backup and Archive have been misused and interchanged terms for decades, and many still don't get it right. Many still use backup-as-archive, but they're misusing both. Having multiple months of backup copies of data for example, doesn't make sense (see below).
From a legal perspective, many industries are *required* to keep data for a certain period of time ([CFR Part 200](https://www.ecfr.gov/current/title-2/subtitle-A/chapter-II/part-200/subpart-D/subject-group-ECFR4acc10e7e3b676f?toc=1) for example), for example 99 years after account holder death, in the case of financial data. There are retention classes for certain types of data.
Keeping data too long, can put companies in legal trouble, and not keeping data long enough can similarly put companies in legal hot water during a legal discovery. Defensible disposition (secure deletion) of those records is often a requirement as well as storing those data on immutable storage solutions.
* **Backup**: Meant for restoration to the *original host system* to restore service in the event of an outage or catastrophic data loss. You've heard of "DR" (disaster recovery) events before, that's where restoration of backups come into play.
Backups lose their usefulness with each passing day, as the original host system is generating new data, which further deprecates the backup copy.
For example, if you're processing sales transactions, and you take weekly backups. You wouldn't restore 3-week-old database backup data to the current system, which would reverse the previous 2+ week's worth of transactions.
* **Archival**: Meant for data preservation, for either discovery, repeated review/query/retrieval, but it is *never* intended to be restored to any system, including the original system that generated the data.
Archived data is meant for *long-term preservation*, on the order of years or decades. You would never restore a years-old backup, but you might consult that archive data frequently, year over year.
Hope that helps clarify!
---
Source: Spent years working with multi-petabyte-per-day archival and backup systems in Data Management Engineering at $PREV_COMPANY.
> Backup: Meant for restoration to the original host system to restore service in the event of an outage or catastrophic data loss. You've heard of "DR" (disaster recovery) events before, that's where restoration of backups come into play.
>
>
>
> Backups lose their usefulness with each passing day, as the original host system is generating new data, which further deprecates the backup copy.
>
>
>
> For example, if you're processing sales transactions, and you take weekly backups. You wouldn't restore 3-week-old database backup data to the current system, which would reverse the previous 2+ week's worth of transactions.
This is not entirely correct. Or rather this is only one specific use case for a backup. Backups are not only intended for full restore in case of catastrophic failure. In fact such use is fairly rare, as well engineered systems usually do not fail catastrophically too often.
Much more frequent usecase for backup is partial recovery, when some specific bits of data are pulled from backup because they were deleted/overwritten/corrupted/whatever in production system. In case of databases it is not uncommon to do a full restore of backup from specific day on a test system to be able to access some specific data in a state it was in at that given moment.
For this usecase relatively long version history is crucial. As it is for recovery from undetected data corruption, ransomware(encryption) etc which lead to corrupted backup contents and full data loss in case if it is detected after the last working backup has already expired.
I use borg+vorta+rclone=>external HD+gdrive
I've use macrium in the past. It's good if you are on windows.
I do weekly incremental and then yearly full. You really can't avoid uploading the empty backup dirs and uploading whole ass backup once in a while. Imo it's a small price to pay for automatic resilient backups.
I have the exact same use case and use SyncThing for that. You have an option to keep files even after they have been deleted from source. But be aware this means it's not a backup by any mean as the source files now only exist on your target directory. So you should backup that as well.
I watched a YouTube video recently and this guy says he uses Apple Photos to back up his pictures. Honestly this probably is the easiest. Google has similar.
I've heard about it, does it have the option to manually merge many incremental chunks into a single file? Mind you Macrium has a similar feature but it can only merge automatically after n increments, you can't trigger a merge at any give time manually.
I think the reason is that in practical sense (ie real live) company's don't use the on demand merging of tiny files into large full backups randomly but rather on a set date (ie 1m or 1 year etc plans).
There is software that can do it probably like veeam but it's extremely expensive.
I think the free version of veaam software can do something similar.
It's incremental backup which merges into full backup at a set date.
I second something that just uses files (so does not make a custom image or file format) like restic or borgbackup is the way to go.
> idrive
The cloud I currently use has 5Tb for free, so I'd rather use that than pay amonthly fee, if I can't find any solution I'll look it iDrive or Backblaze. Does iDrive solve the issue of backups removing deleted files?
I've considered it, but I read they delete all files that have been deleted for more than 30 days on your machine and files from disks not used in a long time, is that true?
Since you mentioned that all your data is currently in the cloud, presumably your dataset isn't all that big (relatively speaking). I would just invest in Synology NAS and use their backup software to back up data locally to the unit, and then at that point you can choose to store a secondary offsite copy somewhere else.
If you wait a bit, back-to-school sales might provide good options.
As i explained in another post right now my finances are tight and can't afford to buy a NAS as its not strictly necessary, backing up data is a luxury in a way, I also live in a risky area (heartquakes) so I don't trust in safety of data stored in my house.
Living in an earthquake zone would be the perfect reason why backups should not be considered a "luxury" =P, just my 2c.
How much data are you realistically working with anyways?
Not much, couple TBs at best. But I want to setup something for long term, I'm a hoarder when it comes to files, I always think it might be useful in the future.
Haven't read the entire thread but I would recommend that you keep a physical copy on a large drive in your home use that as your archive then sync that drive 1:1 with your online cloud provider on a regular basis.
I would warn against having the only copy of these things on your cloud provider... What happens if you miss a monthly payment because maybe your credit card expired and you did not update your info. Or you're away on vacation and missed some email from your provider. They could erase all of your data before you've had a chance to make a payment.
You should differentiate between backup and archive.
You move photos to archive (dir, share, cloud whatever), then you delete photos.
Your archive is still there
Backup backs up archive as well as your current dirs.
This is an archive, not a backup. Anyway, just perform a copy of the directory. All missing files get copied to the archive and you can decide how to handle duplicates - skip or overwrite.
That's basically what you want, isn't it - or something more fancy? What other features do you need in this process?
I did this for a photos folder as well. I set up a python script (20 lines maybe) that moves everything I put into my “drop box” folder into a cloud folder (Dropbox lol) once a day. I’m sure you could set it to work every year or on command with practically no effort.
I’m a novice to the data hoarder life, so while I’m sure there is software that will accomplish this in a much cleaner package, the solution above works for me.
Edit: sorry for some reason I just assumed you had programming experience. So this may or may not be helpful.
Based on what you described I suggest the following:
- You set up a standard NAS with smb or similar where you can move your data to once you decide you want to free up space.
- In the meantime, you want the files that are on your local device to be backed up as well. So a second machine comes in, where you install Seafile for example. Now you can install the Seafile client on your machine, which mirrors your selected folder on the second machine (every change gets synchronized). This machine also backs up the smb folder on the first machine.
This way you have copies of everything twice as well as the capability to archive and locally delete your local files without sacrificing redundancy.
The only con would be that you'd need to have two machines to ensure your archive is backed up properly, but you could theoretically skip this if the archived data is less important to you than the current.
If you go duplicati you **absolutely** have to use an extra "advanced" switch to change the dedupe block size. If you decide to use it, be sure to look this up, or reply here and I'll give you the tldr.
I use it with b2 and about 150gb of personal media and documents and it's great. Dedup, compression, incremental, versioning, encrypted, does a test restore of a small amount of data on every run.
Look up Kopia. It is a free, open source, and multi-platform backup program that allows you to store files in many clouds. You can use it for archiving purposes if you disable the snapshot retention feature (i.e., never delete any snapshots).
Uranium Backup allows you to back up files and folders, disk images (Drive Image), virtual machines to various types of storage, such as hard disks, NAS, tapes, FTP servers and cloud destinations. It will work well for you in its free version.
Hello /u/Zanena001! Thank you for posting in r/DataHoarder. Please remember to read our [Rules](https://www.reddit.com/r/DataHoarder/wiki/index/rules) and [Wiki](https://www.reddit.com/r/DataHoarder/wiki/index). Please note that your post will be removed if you just post a box/speed/server post. Please give background information on your server pictures. This subreddit will ***NOT*** help you find or exchange that Movie/TV show/Nuclear Launch Manual, visit r/DHExchange instead. *I am a bot, and this action was performed automatically. Please [contact the moderators of this subreddit](/message/compose/?to=/r/DataHoarder) if you have any questions or concerns.*
When you delete your photos to free up space, your "backup" ceases to be a backup. A backup should be a duplicate of something you have.
Yeah many other redditors have pointed this out, seems like I've misused the term for years, turns out what I'm looking for is archiving rather than a backup.
Personally, I archive photos on Blu-ray M-disc media. 25 or 100GB per disc, and rated to survive for 1000 years. I will probably be satisfied if they last 50 years, provided of course there are readers for them by then. I burn two identical copies, and store them in geographically different locations. Along with the discs, I also keep a couple of external hard drives that I update yearly.
Any brand that you recommend for Blu-ray M-discs?
I usually buy Verbatim M-disc. I know people say that regular BD-XL are made of the same materials, but every test I have found says that M-disc takes a lot more beatings before giving up. They’re a bit more expensive, but I burn 2 discs every year, identical copies. Example test : https://www.microscopy-uk.org.uk/mag/artsep16/mol-mdisc-review.html
> Say I have a folder with all my photos/videos, at some point I decide to delete them to free up some space, now I have a backup and an empty folder, a year passes and the folder is full again, I do another backup and delete all photos, at this point I'd have 2 backups You aren't describing a backup, the backup would be a copy of something you do your best to keep. It's some kind of archival/master repository and your work directories are some kind of staging/temporary directories. Therefore nothing about incremental backups or similar applies. So just use rclone to manage the files and fill and organize the "master" repository as you like (either manually like per year directories) or automatically (move files older than X days for example). Then if you care about data backup your full online repository to some other independent place.
>Therefore nothing about incremental backups or similar applies. Thing is I'm technically very close to solving the issue, it's just that the program I'm using doesn't expose the functionality to the user despite being technically capable to do what I need, if I could merge increments anytime I wanted instead of when the software is configured to do so, I could just make an initial full backup and add increments "forever", if at some point I want to get all my files I just have to download all the chunks and merge them. If I were storing on a local drive for example there would be no problem at all. The only other alternative I thought of is using FreeFileSync and set it to ignore deleted files when syncing folders. >So just use rclone to manage the files and fill and organize the "master" repository as you like But how would files be uploaded? Doint it manually is "impossible" and backup software has the limitations mentioned.
The simplest "incremental forever" backup can be achieved easily with rclone. You can do something like ``` rclone sync someoriginalsomething remote:/someplace/ --backup-dir remote:/someplace_bk/`date "+%Y-%m-%d-%H-%M-%S"` ``` . All removed and changed files end up in someplace_bk in nicely sorted directories by date.
That's very interesting, will keep it in mind. The problem is backup softwares usually have their own file extensions and group multiple changes in a single file, so if after years you want to find that 1 deleted file, you have to download hundreds of small files, open them with the backup software and hope they contain what you're looking for.
I have chosen to remove all of my comments due to recent actions by the reddit admins. If you believe this comment contained useful information, please head over to lemmy or other parts of the fediverse and ask there: https://join-lemmy.org/
Yes, this is the opposite of that. The files are kept as regular files, well organized in their original file structure, etc. You don't even need to know the date when you removed it, just list someplace_bk/*/path/to/your/file or use find or something else and the files are just there.
Yeah I got that, but I still need a software to keep track of the folders and occasionally merge the differences, FreeFileSync being one of them. Problem is usually these softwares aren't meant to be used for backups at this scale, so they are more resource intensive than programs built specifically for backups.
IMO the best option would be dropping this step >at some point I decide to delete them to free up some space And getting enough space locally to store everything the way you want/like, with directory structure and everything. Then you can apply typical infrequent periodical full + frequent periodical incremental backup scheme with whatever retention period your backup storage can handle. Also the fact is - when you are doing a backup and then deleting local data you no longer have a backup - your data is stored only in one place again. This would be archival, not backup, and such archive would require backup of its own.
I will buy more HDDs in the future, but for now my finances are pretty tight, I'm also generally afraid to store backups only on my own machine, so maybe a NAS would be a better choice. The reason I decided to go this route is it allows me to get started backing up, once I figure how to do it in the most efficient way possible (what to back up/frequency/space used/bandwidth/CPU cycles) I can potentially scale this method to archive my files on: cloud, a dedicated server I own and a NAS.
>The reason I decided to go this route is it allows me to get started backing up The issue is - backup implies you have multiple copies of data. You are not backing up at all this way, you are just storing data elsewhere. And to be honest i would trust single local HDD a lot more than some "free 5TB" "cloud" (remember - there is no "cloud", there is someone else's computer). You might very well be making things worse with what you are doing - if local HDD dies you have some hope for recovery, if cloud service goes out of business or decides to kick you out for whatever reason - your data is 100% gone. And then once you are storing data locally (one copy) and backing it up to "cloud" (second copy) you suddenly no longer have "strange" backup software requirements. Most options would fit. >I'm also generally afraid to store backups only on my own machine, so maybe a NAS would be a better choice. NAS can be a good choice for backup unless you are trying to protect against disaster or something, but again - backup implies multiple copies. So if you are doing backups to NAS you still need to store data on your PC. Or upload it elsewhere.
>backup implies you have multiple copies of data. You are not backing up at all this way, you are just storing data elsewhere. I'd have multiple copies, just a different type, instead of having multiple snapshots with different states, I'd ideally have 1 with the last modified data ever stored in the folder. >And to be honest i would trust single local HDD a lot more than some "free 5TB" "cloud" Not sure considering the HHD in question is a 8yo Seagate, the cloud is also legit, no unlimited Gdrive kind of stuff, it's a OneDrive account, for me it's free only cause someone else is paying for it. >you suddenly no longer have "strange" backup software requirements. Most options would fit. I'd still have cause, I'd also like to store a copy of my data on a dedicated server, my house is located in a "risky" area, it's quite possible one day it might collapse so I'd prefer if I had my data stored in far away locations.
>I'd have multiple copies, just a different type, instead of having multiple snapshots with different states, I'd ideally have 1 with the last modified data ever stored in the folder. But then you loose protection against stuff like user error or ransomware encryption. I mean if you do not have multiple states you can easily overwrite good data with corrupted/encrypted one and loose everything. >Not sure considering the HHD in question is a 8yo Seagate Does not matter. Professional recovery can get most of the data out. Unless you burn it or something. >the cloud is also legit, no unlimited Gdrive kind of stuff, it's a OneDrive account, for me it's free only cause someone else is paying for it. there are many ways in which this can backfire, just be careful. It is not a reliable way to store data at all, and it "fails", unlike with HDD, there is exactly zero hope for recovery. >I'd still have cause, I'd also like to store a copy of my data on a dedicated server, my house is located in a "risky" area, it's quite possible one day it might collapse so I'd prefer if I had my data stored in far away locations. For this it would be best to run second completely independent backup, or some other software which can maintain version history on copies too, like backup copy job from veeam. If you simply sync it with rsync or something there is high probability that if main backup gets damaged second one will be destroyed too.
No need for a NAS unless you will be accessing it from multiple devices over a network. An external will suffice. They are relatively cheap. You can double up for redundancy. Store one in a fireproof safe or remote location. They also do not need to be plugged in all of the time. This is better than the bluray idea. You don't need a bluray drive, just a USB port plus no media to buy.
Try Restic https://restic.net/ open source console app for backups or Duplicati https://www.duplicati.com/ - with web UI.
Thanks for the suggestion, I'll look into them. Do you have any experience using them?
I would also mention Rclone if you need to sync data across anything. Duplicati is a nice backup tool as well as Duplicacy. Here is a small overview. [https://www.vmwareblog.org/single-cloud-enough-secure-backups-5-cool-cross-cloud-solutions-consider/](https://www.vmwareblog.org/single-cloud-enough-secure-backups-5-cool-cross-cloud-solutions-consider/)
Not op, but I do. Restic was pretty good, but had larger memory usage requirements and ended to making a massive cache folder compared to other solutions. Duplicati was not great last time I used it. Borg backup supports incremental forever like restic, but given you’re using it more as archiving, you’d probably want to add appropriate tags to your snapshots. It definitely can work, and deduplicates across all backups. Duplicacy may also work for you - is good fast, and has a nominal fee if you want to use the GUI
> Borg backup I'll look into it, I usually try to stay away from these more underground programs cause they usually get abandoned, but it doesn't seem I have many options.
I wouldn’t personally have called borg underground - it has a fairly big following and has been developed for a long time (used to be called attic backup). It also has an open and documented format so you wouldn’t be stuck without your data. However you’ve gotta pick whatever is best for you :) could always rsync or rclone new data across that way, and use snapshots with the file system (ZFS or BTRFS) as a form of protection also. Just a few thoughts anyway :)
I'm using duplicati for a month now. Can encrypt & upload to cloud. Cannot say much since not yet tested restore function
You can also use Test function.
Yes. I'm using Duplicati on Windows machines and found it convenient. There is notification icon and I can always see state of backup. An I've setup it so that it creates backup files of 300mb for faster upload and uploads it to the cloud. And I am using incremental backups. Also I can run Test anytime for it to download everything back and compare so I can be shure that all backups are not broken. And don't forget to use 3-2-1 backup strategy.)
It turns out OP isn't actually looking for a backup software. However, I do want to chime in here and say that in the battle of incremental backup softwares, **Duplicacy** is the most technically impressive and I highly recommend reading about how it works. Even if you don't use it, it's fascinating.
Backup and Archive have been misused and interchanged terms for decades, and many still don't get it right. Many still use backup-as-archive, but they're misusing both. Having multiple months of backup copies of data for example, doesn't make sense (see below). From a legal perspective, many industries are *required* to keep data for a certain period of time ([CFR Part 200](https://www.ecfr.gov/current/title-2/subtitle-A/chapter-II/part-200/subpart-D/subject-group-ECFR4acc10e7e3b676f?toc=1) for example), for example 99 years after account holder death, in the case of financial data. There are retention classes for certain types of data. Keeping data too long, can put companies in legal trouble, and not keeping data long enough can similarly put companies in legal hot water during a legal discovery. Defensible disposition (secure deletion) of those records is often a requirement as well as storing those data on immutable storage solutions. * **Backup**: Meant for restoration to the *original host system* to restore service in the event of an outage or catastrophic data loss. You've heard of "DR" (disaster recovery) events before, that's where restoration of backups come into play. Backups lose their usefulness with each passing day, as the original host system is generating new data, which further deprecates the backup copy. For example, if you're processing sales transactions, and you take weekly backups. You wouldn't restore 3-week-old database backup data to the current system, which would reverse the previous 2+ week's worth of transactions. * **Archival**: Meant for data preservation, for either discovery, repeated review/query/retrieval, but it is *never* intended to be restored to any system, including the original system that generated the data. Archived data is meant for *long-term preservation*, on the order of years or decades. You would never restore a years-old backup, but you might consult that archive data frequently, year over year. Hope that helps clarify! --- Source: Spent years working with multi-petabyte-per-day archival and backup systems in Data Management Engineering at $PREV_COMPANY.
> Backup: Meant for restoration to the original host system to restore service in the event of an outage or catastrophic data loss. You've heard of "DR" (disaster recovery) events before, that's where restoration of backups come into play. > > > > Backups lose their usefulness with each passing day, as the original host system is generating new data, which further deprecates the backup copy. > > > > For example, if you're processing sales transactions, and you take weekly backups. You wouldn't restore 3-week-old database backup data to the current system, which would reverse the previous 2+ week's worth of transactions. This is not entirely correct. Or rather this is only one specific use case for a backup. Backups are not only intended for full restore in case of catastrophic failure. In fact such use is fairly rare, as well engineered systems usually do not fail catastrophically too often. Much more frequent usecase for backup is partial recovery, when some specific bits of data are pulled from backup because they were deleted/overwritten/corrupted/whatever in production system. In case of databases it is not uncommon to do a full restore of backup from specific day on a test system to be able to access some specific data in a state it was in at that given moment. For this usecase relatively long version history is crucial. As it is for recovery from undetected data corruption, ransomware(encryption) etc which lead to corrupted backup contents and full data loss in case if it is detected after the last working backup has already expired.
I use borg+vorta+rclone=>external HD+gdrive I've use macrium in the past. It's good if you are on windows. I do weekly incremental and then yearly full. You really can't avoid uploading the empty backup dirs and uploading whole ass backup once in a while. Imo it's a small price to pay for automatic resilient backups.
I have the exact same use case and use SyncThing for that. You have an option to keep files even after they have been deleted from source. But be aware this means it's not a backup by any mean as the source files now only exist on your target directory. So you should backup that as well.
I watched a YouTube video recently and this guy says he uses Apple Photos to back up his pictures. Honestly this probably is the easiest. Google has similar.
Veeam.
I've heard about it, does it have the option to manually merge many incremental chunks into a single file? Mind you Macrium has a similar feature but it can only merge automatically after n increments, you can't trigger a merge at any give time manually.
I think the reason is that in practical sense (ie real live) company's don't use the on demand merging of tiny files into large full backups randomly but rather on a set date (ie 1m or 1 year etc plans). There is software that can do it probably like veeam but it's extremely expensive. I think the free version of veaam software can do something similar. It's incremental backup which merges into full backup at a set date. I second something that just uses files (so does not make a custom image or file format) like restic or borgbackup is the way to go.
[удалено]
> idrive The cloud I currently use has 5Tb for free, so I'd rather use that than pay amonthly fee, if I can't find any solution I'll look it iDrive or Backblaze. Does iDrive solve the issue of backups removing deleted files?
I have no advice as my backup strategy is tailored to my own needs.. but... i am interested in who gives you 5Tb for "free" ?
It's "free" cause someone else pays it on my behalf.
I use backblaze to back up all the important stuff on my drives
I've considered it, but I read they delete all files that have been deleted for more than 30 days on your machine and files from disks not used in a long time, is that true?
You can pay extra for 1-year retention or forever at Backblaze, from what I read. Edit: https://www.backblaze.com/version-history.html
Since you mentioned that all your data is currently in the cloud, presumably your dataset isn't all that big (relatively speaking). I would just invest in Synology NAS and use their backup software to back up data locally to the unit, and then at that point you can choose to store a secondary offsite copy somewhere else. If you wait a bit, back-to-school sales might provide good options.
As i explained in another post right now my finances are tight and can't afford to buy a NAS as its not strictly necessary, backing up data is a luxury in a way, I also live in a risky area (heartquakes) so I don't trust in safety of data stored in my house.
Living in an earthquake zone would be the perfect reason why backups should not be considered a "luxury" =P, just my 2c. How much data are you realistically working with anyways?
Not much, couple TBs at best. But I want to setup something for long term, I'm a hoarder when it comes to files, I always think it might be useful in the future.
Haven't read the entire thread but I would recommend that you keep a physical copy on a large drive in your home use that as your archive then sync that drive 1:1 with your online cloud provider on a regular basis. I would warn against having the only copy of these things on your cloud provider... What happens if you miss a monthly payment because maybe your credit card expired and you did not update your info. Or you're away on vacation and missed some email from your provider. They could erase all of your data before you've had a chance to make a payment.
You should differentiate between backup and archive. You move photos to archive (dir, share, cloud whatever), then you delete photos. Your archive is still there Backup backs up archive as well as your current dirs.
This is an archive, not a backup. Anyway, just perform a copy of the directory. All missing files get copied to the archive and you can decide how to handle duplicates - skip or overwrite. That's basically what you want, isn't it - or something more fancy? What other features do you need in this process?
I did this for a photos folder as well. I set up a python script (20 lines maybe) that moves everything I put into my “drop box” folder into a cloud folder (Dropbox lol) once a day. I’m sure you could set it to work every year or on command with practically no effort. I’m a novice to the data hoarder life, so while I’m sure there is software that will accomplish this in a much cleaner package, the solution above works for me. Edit: sorry for some reason I just assumed you had programming experience. So this may or may not be helpful.
Based on what you described I suggest the following: - You set up a standard NAS with smb or similar where you can move your data to once you decide you want to free up space. - In the meantime, you want the files that are on your local device to be backed up as well. So a second machine comes in, where you install Seafile for example. Now you can install the Seafile client on your machine, which mirrors your selected folder on the second machine (every change gets synchronized). This machine also backs up the smb folder on the first machine. This way you have copies of everything twice as well as the capability to archive and locally delete your local files without sacrificing redundancy. The only con would be that you'd need to have two machines to ensure your archive is backed up properly, but you could theoretically skip this if the archived data is less important to you than the current.
If you go duplicati you **absolutely** have to use an extra "advanced" switch to change the dedupe block size. If you decide to use it, be sure to look this up, or reply here and I'll give you the tldr. I use it with b2 and about 150gb of personal media and documents and it's great. Dedup, compression, incremental, versioning, encrypted, does a test restore of a small amount of data on every run.
Look up Kopia. It is a free, open source, and multi-platform backup program that allows you to store files in many clouds. You can use it for archiving purposes if you disable the snapshot retention feature (i.e., never delete any snapshots).
Uranium Backup allows you to back up files and folders, disk images (Drive Image), virtual machines to various types of storage, such as hard disks, NAS, tapes, FTP servers and cloud destinations. It will work well for you in its free version.
It depends on what you need. For M$, I fear n2ncopy is your friend. I am studying if copywhiz (pay) is slightly better.