Modern file backup options: cloud? / HD?

[DaveC426913]

TL;DR Summary

I want to stop losing my precious projects. I need a reliable way to backup files for me, and friends & fam

I thought I had a long-term backup of my important stuff when I bought a 1Tb Seagate External Drive. It died and took years of projects and memories with it.

Running Win10 on a laptop.

I think there are three options:

1. Another (better) external drive
2. Some newfangled solid-state external drive
3. Cloud storage

Some considerations:

• I want to ensure it covers mine and my wife's files - (and, in an ideal world, daughter-in-law-at-separate-residence). Whether that means one, two or three devices/services.
• I think I'll need to set up an auto-backup service for my wife. I personally didn't need one, but I'm more conscientious about securing my important files. But maybe I'll let a computer do something for me that I can do myself. (Big step there.)
• I don't want to spend a half hour every month watching progress bars push my files around. (I guess that's auto backup during off hours)
• I do have my own webhost and domain, if that makes any diff. There is storage there. I'm just not sure it's a serious, robust consideration (I've had bad experiences with webhosts).

Anyway, I'm looking for advice and personal opinions about what to go with.

 

[jedishrfu]

The best strategy is multi format backup. I save stuff to sticks, cds/dvds, external drives and the cloud.

if you were paranoid then you might use multiple cloud backup as well as multi drive raid backup with backups saved in bank vaults at multiple locations.

 

[DaveC426913]

Nope, not that paranoid. Have to have a balance between robustness and effort/convenience.(Forgot to mention CDs. They're pretty robust, long-term, but the routine burning process is prohibitive.)

 

[Vanadium 50]

Here's what I do.

I have a central file server. Format is ZFS, which does "snapshots". I take one every 15 minutes, keeping 4, one every hour, keeping 48, one every day, keeping 31, one every week, keeping 7, and one every month, keeping 60. Also, ZFS mirrors every disk, and every two weeks I check every byte for integrity.

Weekly, I back up my desktop PC to the server.

Daily, I backup the central server to an external hard disk using rsync. This typically takes 30 seconds, unless I have recently done a weekly backup, in which case it takes 20 minutes. I keep the external drive in ext4 format. The idea is that if I bork my ZFS pool, I can mount the backup while I figure out went wrong.

My biggest risk is correlated by having everything in one place. Pre-Covid, I kept an external drive at work and another at home and swapped them weekly.

The downside? Cost. Every byte is stored no fewer than 3 times. That costs money. On the plus side, I have never lost a byte.

 

[jedishrfu]

Searching on tape backup strategy I found this article:

https://www.techrepublic.com/article/tech-tip-determine-your-tape-backup-strategy/

I remember years ago there was a work strategy of doing incremental backups and periodically a full backup and there was a scheme to use a limited number of tapes in a clever way to maximize retention across multiple tapes which could apply today with any kind of media. Sadly, I can't remember / didn't actually know the full strategy being a lowly programmer at the time.

However, this Tower of Hanoi strategy looks like the one my company used:

https://www.recordnations.com/articles/tape-rotation-2/

 

[nsaspook]

You can never be too paranoid about data loss. I don't trust 'cloud' backup.

My main personal work stations are surplus diskless HP ProLiant DL360 G7 1U 12 core 64-bit Servers running 64-bit Linux Debian remote boot over the network to a main 12 core DL360 G7 with a 8 SAS drive raid and 4 port 1Gbit ethernet for remote NFS mounts to the workstations.

A typical surplus under $200 system: HP Proliant DL360 G7 12 Core Server [ 96G PC3 RAM - 4TB HD - 2x 2.40GHz ]
https://h20195.www2.hpe.com/V2/Getdocument.aspx?docname=c04284501

A physically (workshop on the house lot) separate computer system uses Bacula to backup the main server and other computers to internal 10TB mirrored drives and external 10TB USB-C raid storage pools. It automatically selects incremental and full backups.

 

[jedishrfu]

Here’s a Smarter Everyday video on how he improved his storage system

 

[FactChecker]

Summary:: I want to stop losing my precious projects. I need a reliable way to backup files for me, and friends & fam

I thought I had a long-term backup of my important stuff when I bought a 1Tb Seagate External Drive. It died and took years of projects and memories with it.

If that drive was a backup, then you must still have your work on the working drive. If that is the only drive where the files existed, it is not a backup. I recommend periodic backups of your work on a separate drive, which is stored in a separate location. If you want more security, you could set up a RAID system where two drives mirror each other and notify you if one of them has problems.

Running Win10 on a laptop.

I think there are three options:
Another (better) external drive

Being external does not make it a backup drive unless you run backups periodically and keep it in a different location.

Some newfangled solid-state external drive

Same

Cloud storage

Cloud storage maintained by a reputable company will tend to be much safer than the typical system run by an amateur. But I do not know what it costs or how much data you have.

 

[Wrichik Basu]

You can use more than one external drive for storing files that you don't want in your PC's hard disk. Two or more external drives are less probable to fail at the same time, so you will always have a copy of your files on at least one drive.

For my programming projects, I always initialize Git and push them to GitHub (in a private repo where necessary). I can pull the repo even if I lose my local folder.

You can use a Raspberry Pi to create a NAS with an external HDD. There are a lot of tutorials to guide you on the process.

 

[MikeeMiracle]

One backup is never enough if you really don't want to lose anything. Anything physical has the potential to fail, especially external HDD's so you need a multi backup approach. You also need online and offline backups to guard against online and offline threats.

I would use a cloud service for online backups and probably an external SSD for occasional offline backups.

Using an online cloud service prevents you loosing data by suffering a hardare / disk failure or worse, like say your house gets struck by lightning that blows all your electronics or your house burning down. Don't count on keeping all your data just on your premisis.

It does not however help of you get malware which then encrypts all your data no matter where it's kept, this is where offline backups come in that are not connected to anything and only connect while the backup is occurring. This can be an external HDD / SSD you plug into backup your data once per week.

 

[pbuk]

Microsoft 365 Family. You and your wife (not sure about licensing for daughter-in-law at different address) each get 1TB of cloud storage that synchronizes in the background transparently. Also other benefits.

Not sure what kind of 'projects' you are talking about: if it is software projects then that doesn't work well with cloud backup (large number of file changes every time you switch branches, transient build artifacts, 3rd party libraries) so GitHub/GitLab/Bitbucket is the answer.

 

[anorlunda]

If you want your backups to last a lifetime, don't forget that they must be migrated to new technology as it becomes available. You can't count on Windows, or Unix variants, or file formats like JPG or PDF or DOC being around forever.

Library science deals with those problems. They found that digital data storage can be more difficult to archive than ancient scrolls, painting, and writings.

I think back to the first backups I made. They were paper tape. The tape might still be in good condition, but try finding a paper tape reader today, and try to find a computer that can run those executable files and source programs (with their dialects and OS dependencies).

We also had magnetic tapes in those days, but they had the same problem. Lots of luck finding a way to read those tapes today.

Migration might include changing file formats and rewriting source programs.

 

[Vanadium 50]

My main personal work stations

Aren't they loud? Anway, back on topic:

One thing to consider with cloud backup is the time to back up and the time to restore. If you have 10 Mb/s ethernet and want to backup or restore 8 TB, that's 80 days up and 80 days down. Even at 100 Mb/s, it's more than a week.

Some commercial clouds will FedEx you an external USB drive if you need to get your data back. Of course you pay for this.

 

[DaveC426913]

If that drive was a backup, then you must still have your work on the working drive. If that is the only drive where the files existed, it is not a backup.

Well of course, I didn't go looking on my backup for filesvuntil my laptop crashed. That's when I found the external drive had stopped working

But you do make a point, I was using it as an archive as well, to make room on my computer.

I hadn't really been making that distinction until you pointed it out

 

[DaveC426913]

Not sure what kind of 'projects' you are talking

photos, Photoshop files, documemts , not code

 

[Vanadium 50]

We also had magnetic tapes in those days, but they had the same problem. Lots of luck finding a way to read those tapes today.

There are companies that will transfer 6250 tapes to LTO. Maybe even a USB stick. Paper tape, well, you're on your own, I think.

 

[DaveC426913]

Question: given the choice of cloud storage and one external drive, which would you trust more, long-term?

 

[anorlunda]

Question: given the choice of cloud storage and one external drive, which would you trust more, long-term?

Define long term. 2 years? 20 years? 200 years?

 

[FactChecker]

photos, Photoshop files, documemts , not code

I keep a large collection (~150 GB) of music and photos on multiple external USB drives. I don't store new stuff to them except once in a while and consider them as long-term archival rather than regular backups. It is a problem to keep them up to date and identical, so I accept some losses if the worst happens. I do regular backups to a RAID system and to another USB drive.

 

[DaveC426913]

Yes this is just personal use stuff for me, So I accept a certain amount of rot.

My wife's stuff is business stuff though mostly documents.

Email is another concern for her

 

[FactChecker]

Well of course, I didn't go looking on my backup for files until my laptop crashed. That's when I found the external drive had stopped working.

I understand. You are not alone there. I have horror stories. I occasionally do spot checks to see if it looks like files are on the backup drive. I hope that your data can be recovered. Not all broken backup drives are completely broken.
PS. My horror stories are the reason that I have so many backup drives, RAID systems, etc., and I still don't feel completely safe. Good luck.

 

[pbuk]

photos, Photoshop files, documemts , not code

Ok, these are fine for cloud backup.

Question: given the choice of cloud storage and one external drive, which would you trust more, long-term?

Cloud of course. If your external drive breaks then you are the only one interested in fixing it. If OneDrive goes down there are resources of USDbns (and GBPbns, and EURbns) interested in fixing it.

 

[nsaspook]

Aren't they loud? Anway, back on topic:

The disk-less server class workstations with upgraded video cards are very quiet at idle with a proper Linux fan-control daemon. The main servers, online UPS and house networking on are an old DEC rack in the garage so noise is not a problem there.


I've mainly used physically redundant hard-drive based backup systems with limited offsite backup to sites like github for sharing ongoing projects.

 

[phinds]

I got so pissed off at myself about 20 years ago for losing an important file that I wrote myself a backup program and I've improved on it some since then. I won't lose any files even if the house burns down (well, I might lose a couple of days work).

I just plug in my external drive, click on one icon and I'm done (the computer isn't done but who cares). When it finishes I swap that drive out with the same kind of drive that I keep in the trunk of my car (I used to take one to work with me but I'm retired now so "off-site" is the trunk of my car. If I'm adding a lot of files I might do that as often as every day or two but usually it's about once a week.

Every year or so I swap one of them out with a third that my son keeps (WAY off-site).

The program only backs up new files and files that have changed. Optionally, it will delete files that are no longer on the internal hard drives (it backs up various files from several drives).

None of that is much help to you, Dave, I know, but my point is that I think that if you want a reliable backup system you need at least 2 external drives (I don't use the cloud) and my preference is for 3.

 

[FactChecker]

I think you can get some secure cloud backup space essentially for free. That is certainly the most secure. I don't know how much data they allow before you have to pay for more storage space. Being an old-timer, I want my backups to be clutched in my sweaty little hand. ;-)

 

[Vanadium 50]

You can backup a lot for free. If you want to restore...that'll cost you.

 

[phinds]

I back up about about 500,000 gigs so even if it were free storage the download for a backup would be a killer.

I've written a "delta site mirror" program for my wood ID site (200,000+ JPG and HTML files) that I could expand to my whole backup but I would not be happy w/ the amount of time it would take so I use the external drives.

 

[DaveC426913]

I back up about about 500,000 gigs so even if

Did you really mean a half a petabyte?

 

[Vanadium 50]

Did you really mean a half a petabyte?

Wood is serious business.

 

[phinds]

Did you really mean a half a petabyte?

No, I meant what I said. Half a terabyte.

 

[Vanadium 50]

I think there are some extra zeros there. Half a terabyte is only 500 GB.

 

[FactChecker]

I back up about about 500,000 gigs so even if it were free storage the download for a backup would be a killer.

I've written a "delta site mirror" program for my wood ID site (200,000+ JPG and HTML files) that I could expand to my whole backup but I would not be happy w/ the amount of time it would take so I use the external drives.

I think that the cloud system would have to be considered the mass storage and only download the data that is in use currently. That is usually a very small percentage. I would leave the mass backups and recoveries to the cloud data administrators.

 

[DaveC426913]

No, I meant what I said. Half a terabyte.

Er Nope.

 

[phinds]

I think there are some extra zeros there. Half a terabyte is only 500 GB.

Er Nope.

 

[pbuk]

500GB is not too much for cloud backup, and one of the advantages of some systems is that you can restore incrementally and selectively.

If you need to rebuild your local copy (perhaps when changing main computers), select the files you want to work on right now and they will appear almost immediately. Leave it running overnight and when you get up, all or most of it will already be done - anything you need that is not there yet you can pull forward at the click of a mouse.

Of course if your internet connection is limited you will need another solution.

 

[MikeeMiracle]

I currently have 1.2TB in cloud backup. I guess the term "backup" is a little misleading.

Traditionally when we think of backups using backup software, they backup to tape or a "backup file" which contains all of the other files rolled into 1 big file. You need the backup software that created it to extract individual files form it. While you could backup using this method and get the backup software to sync to the cloud I find it makes little sense.

A better way of looking at it is "cloud synchronisation." I used Tresorit as my cloud provider, using this software allows you specify which folders on your computer you wish to "backup" and it then synchronises the files and folders in those locations with the cloud. Whenever I make a change to a file or folder it is automatically synchronised up to the cloud within 30 seconds.

I find this "synchronisation" method more useful. I can have multiple computers connected to this service and I can choose to synchronise any/all or none of the folders with each of my devices.

I have a desktop with plenty of storage which synchronises all it's data this way. I also have a laptop with a 256GB drive. If I wish to access a file on the laptop I just download it, edit it and re-upload it again. This file is then automatically synchronised to the desktop. It's all very seamless just like having a local file server you synchronise to.

On the privacy side, the provider I use called Tresorit is very secure. All your data is encrypted locally on your device and only this encrypted data is sent to the cloud. The encryption uses your password to create a unique encryption key and only that key can be used to decrypt your data. All Tresorit can see is that I have an account and they can see the physical amount of space my data is using but that's about it. There is no "master encryption key" they an use to decrypt and view your data. This means they cannot decrypt your data even if they wanted to so its safe from even security services. The downside to using this method of local encryption is that they also have no way of recovering your data if you forget your password so the onus is very much on you to remember it.

 

[FactChecker]

I currently have 1.2TB in cloud backup. I guess the term "backup" is a little misleading.
...
A better way of looking at it is "cloud synchronisation." I used Tresorit as my cloud provider, using this software allows you specify which folders on your computer you wish to "backup" and it then synchronises the files and folders in those locations with the cloud. Whenever I make a change to a file or folder it is automatically synchronised up to the cloud within 30 seconds.

Exactly. You are using the cloud as your main mass data storage device and are completely freed from the backup responsibility and knowledge. It is really difficult to run a good backup system. That is the job of a system administrator. The cloud device holding one copy of your data might fail and be replaced without you even being aware of it.

 

[MikeeMiracle]

I am a system admin, the way I do it is more of what its called a "Business Continuity / Diaster Recovery (DR)" solution as the data is always online and live from any location.

The MASSIVE mistake people make is to confuse DR with backups and think they are on in the same thing, they are not. My DR solution does not protect me from being hit by malware which encrypts all the files as the encrypted files would get uploaded also. For that you need a backup, i.e an OFFLINE solution also, the simplest of which is an external drive you duplicate your files to.

This is what I meant by having both ONLINE and OFFLINE solutions. It's the only way to truly be protected from data loss.

Oh and periodically check your backups to make sure they are valid and restorable, there are so many stories of companies backing up data and not testing if they can actually restore it until the have to and find out their backups are useless for whatever reason.

 

[DaveC426913]

Oh and periodically check your backups to make sure they are valid and restorable

Tru dat.

 

[rcgldr]

My ISP throttles upload speed (< 20Mbps), so cloud backup isn't a reasonable option for me, at least not for the collection of videos I have. I have 2 desktops with multiple drives and each are multi-boot (Win XP, Win XP X64, Win 7 Pro 64 bit, one of them also Win 10 Pro 64 bit). I backup entire partitions to folders on other drives (while running on one of the "other" operating systems), and also to a set of external drives that I alternate through, or I do system image backup for Win 7 or Win 10 OS partition. I also backup our smart phones to the desktop drives. Occasionally, I'll swap out drives, using a disk cloning utility to duplicate the drive. I do this very carefully, to make sure I copy in the correct direction.

Some of the free cloud servers offered to me when I bought items like smart phones or a lap top have recently shut down.

I mostly use external 2TB hard drives for backup, and a 1 TB SSD drive for file transfer between systems. My concern with SSD drives is that data is lost if the internal mapping is ever messed up. However, there are large servers that are SSD only, using Raid 6 or Raid 6 variation with triple parity, but those servers will always check for unreported errors from hard drives or SSD.

 

[PAllen]

I use a very simple system which totally saved me once. First, I simply don't trust security on the cloud - it has been hacked, and I don't control it. On my windows 10 systems, I force no usage of cloud at all, for any purpose whatsoever. I simply have external drives that I store far away from their back up target, and back up both data files and system images. When a disk had a catastophic failure, I bought a replacement and restored a system image and did not have to reinstall or reconfigure a single thing. Totally up and running with no loss within a few hours. When I wanted an earlier version of a file, I just used the data file backups.

For me, additional music, photos, emails, etc. are such that I only do backup operations every few days. This of course varies by individual usage.

 

[eltodesukane]

There are 2 kinds of hard drives, those who have failed, and those who will fail.
Generally speaking, you can rely on your hard drive for 3 to 5 years on average.
So any data you want to keep must have 2 copies beside the original, and those copies must be recopied every 3 years, and the hard drives replaced as needed.
Hard drives are not a long term storage solution, unless triplicated and actively monitored, maintained and replaced.

 

[Svein]

There are 2 kinds of hard drives, those who have failed, and those who will fail.

A short answer: RAID 1 (or 5 or 10).

 

[George Keeling]

I use Google "Backup and sync" which costs £20 per year for 100GB. Whenever a file changes it gets copied up to the Google cloud and old versions of the file are kept for about a month. That is useful for accidental file wipe outs etc. When my computer died suddenly two years ago I was able to recover everything. My new computer is Windows 10. There was a problem because Windows Defender and Google Drive (Backup and Sync) seem to be incompatible! I got round this and everything is easy again. but ...

I simply don't trust security on the cloud it has been hacked

Another problem is: say a virus infected my computer and corrupted all the files for ransom and they all got into my Google cloud. To protect against this, or Pallen's hack, I have another old computer also connected to the cloud which I turn on once every week or month. The 100 or so files that have changed then get copied down to that computer. It gets turned off again. I would have to be unaware of ransom attack or hack for this system to fail. I think.

No doubt other cloud systems would do as well.

 

[Vanadium 50]

There are 2 kinds of hard drives, those who have failed, and those who will fail.

That's true for every technology. As the old joke goes "now we're just negotiating the price".

You say 3-5 years for a spinning disk. It's probably more than 10 years for a disk that you spin up only once a year. LTO is said to last 30 years. M-disc claims 1000 years, but obviously this isn't something tested. How good does it have to be? How much money are you willing to invest?

As it happens, I had a drive failure last night. Drive (a Toshiba 4 TB) and controller got into a funky state. This happens every 15000 hours or so with these drives.

Everything worked as it should. In the fortnightly consistency check (which happened at the same time as the backup - which is atypical) I got 1.11M checksum errors. So ZFS faulted the drive, degraded the array, and sent me email. I power cycled the system (the usual cure), the drive recovered, and ZFS proceeded to rebuild the array. No user intervention needed (beyond the power cycle), no loss of data, no loss in capabilities (the backup kept right on going) and a minimal loss in performance.

 

[rcgldr]

Traditionally when we think of backups using backup software, they backup to tape or a "backup file" which contains all of the other files rolled into 1 big file.

Depends on the backup software. Some tape backup apps backup folders and files. Some image backup allow for the image to be accessed as a read-only partition in order to do selective folder | file restore. I actually wrote a similar tape backup program for an Atari ST back in the 1980's (image backup, but show as partition for restore, with the FAT table cached in memory to reduce the number of random accesses).

I have a multi-boot system (XP, XP-X64, Win 7, Win 10), and wrote a folder|file copy program that include security and reparse information that I run from one instance of an operating system to backup another. It works fine for XP and XP-X64, but there's something missing for Win 7 restore. It runs, but I get some access errors running browsers, so I suspect some reparse info in Win 7 that XP isn't picking up. I have separate partitions for the OS and installed program | data, so I can use it for my Win 7 and Win 10 program | data partitions. Currently I'm using system image backups for the Win 7 | Win 10 partitions. I may buy one of the image backup programs that allow for folder | file restore.

 

[rcgldr]

I occasionally do full restores in order to "refresh" the data on my hard drives. I don't know if this would be possible or needed with solid state drives.

 

[Vanadium 50]

I occasionally do full restores in order to "refresh" the data on my hard drives

How do you know you aren't replacing good data with corrupted data?

 

[rcgldr]

I occasionally do full restores in order to "refresh" the data on my hard drives.

How do you know you aren't replacing good data with corrupted data?

I do a folder + file compare after each folder + file backup of a partition. For the image backups, I'll make two of each. I haven't tried comparing image backups yet. I may buy software that creates and checks hashes (like SHA256) keys to verify the backed up files in an image backup.

In my case, I have a multi-boot system, and run the folder + file backup from a different OS than the one being backed up. There are some standalone utilities that do something similar, but I haven't tried any of them yet.

I've also bought new hard drives to replace old ones, using standalone software to clone the drives. Just need to be very careful that you copy in the correct direction.

 

[Tom.G]

Windows at least to have two utilities for file comparison FC.EXE and WINDIFF.EXE. At least WINDIFF (a graphical interface) will also do Directory compares.