datahoarder
Who are we?
We are digital librarians. Among us are represented the various reasons to keep data -- legal requirements, competitive requirements, uncertainty of permanence of cloud services, distaste for transmitting your data externally (e.g. government or corporate espionage), cultural and familial archivists, internet collapse preppers, and people who do it themselves so they're sure it's done right. Everyone has their reasons for curating the data they have decided to keep (either forever or For A Damn Long Time). Along the way we have sought out like-minded individuals to exchange strategies, war stories, and cautionary tales of failures.
We are one. We are legion. And we're trying really hard not to forget.
-- 5-4-3-2-1-bang from this thread
view the rest of the comments
Sounds fine?
Yes: Treat the two enclosures independently and symmetrically, such that you can fully restore from either one (the only difference would be that the one in the safe is slightly stale) and the ongoing upkeep is just:
If I assume a normal incremental backup setup, both enclosures would have a full backup and a pile of incremental backups. For example, if swapped every three days:
The thing taking the backups need not even detect or care which enclosure is plugged in -- it just uses the last incremental on that enclosure to determine what's changed & needs to be included in the next incremental.
Nothing need care about the number or identity of enclosures: You could add a third if, for example, you found an offsite location you trust. Or when one of them eventually fails, you'd just start using a new one & everything would Just Work. Or, if you want to discard history (eg: to get back the storage space used by deleted files), you could just wipe one of them & let it automatically make a new full backup.
Are you asking for help with software? This could be as simple as dar and a shell script.
My personal preference is to tell the enclosure to not try any fancy RAID stuff & just present all the drives directly to the host, and then let the host do the RAID stuff (with lvm or zfs or whatever), but I understand opinions differ. I like knowing I can always use any other enclosure or just plug the drives in directly if/when the enclosure dies.
I notice you didn't mention encryption, maybe because that's obvious these days? There's an interesting choice here, though: You can do normal full-disk encryption, or you could encrypt the archives individually. Dar actually has an interesting feature here I haven't seen in any other backup tool: If you keep a small
--aux
file with the metadata needed for determining what will need to go in the next incremental, dar can encrypt the backup archives asymmetrically to a GPG key. This allows you to separate the capability of writing backups and the capability of reading backups. This is neat, but mostly unimportant because the backup is mostly just a copy of what's on the host. It comes into play only when accessing historical files that have been deleted on the host but are still recoverable from point-in-time restore from the incremental archives -- this becomes possible only with the private key, which is not used or needed by any of the backup automation, and so is not kept on the host. (You could also, of course, do both full-disk encryption and per-archive encryption if you want the neat separate-credential for deleted files trick and also don't want to leak metadata about when backups happen and how large the incremental archives are / how much changed.) (If you don't full-disk-encrypt the enclosure & rely only on the per-archive encryption, you'd want to keep the small --aux files on the host, not on the enclosure. The automation would need to keep one --aux file per enclosure, & for this narrow case, it would need to identify the enclosures to make sure it uses that enclosure's --aux file when generating the incremental archive.)Thank you so much for all the information! I actually simply use rsync to synchronize all my data to whichever backup drive is connected. Are there benefits to using dar instead of rsync? I haven't used dar before. I also haven't dabbled in encrypting my backup... My concern with that would be the additional time it takes to encrypt everything, as well as my fear of losing the ability to decrypt it. I am mostly putting this out there to see if anyone has suggestions for different media I hadn't thought of, or if anyone sees any pitfalls in my plan I hadn't thought of. I'm planning on storing the enclosure in a phoenix datacare 2001 fireproof safe. It's supposed to be able to keep hard drives, CDs, tapes, etc. safe in a fire for over an hour.
The benefit of using something fancier than rsync is that you get a point-in-time recovery capability.
For example, if you switch the enclosures weekly, rsync gives you two recovery options: restore to yesterday's state (from the enclosure not in the safe) and restore to a state from 2-7 days ago (from the one in the safe, depending on when it went into the safe).
Daily incremental backups with a fancy tool like dar let you restore to any previous state. Instead of two options, you have hundreds of options, one for each day. This is useful when you mess up something in the archive (eg: accidentally delete or overwrite it) and don't notice right away: It appeared, was ok for awhile, then it was bad/gone and that bad/gone state was backed up. It's nice to be able to jump back in time to the brief it-was-ok state & pluck the content back out.
If you have other protections against accidental overwrite (like you only back up git repos that already capture the full history, and you
git fsck
them regularly) — then the fancier tools don't provide much benefit.I just assumed that you'd want this capability because many folks do and it's fairly easy to get with modern tools, but if rsync is working for you, no need to change.