Modern file backup options: cloud? / HD?

[Vanadium 50]

I think there are some extra zeros there. Half a terabyte is only 500 GB.

 

[FactChecker]

I back up about about 500,000 gigs so even if it were free storage the download for a backup would be a killer.

I've written a "delta site mirror" program for my wood ID site (200,000+ JPG and HTML files) that I could expand to my whole backup but I would not be happy w/ the amount of time it would take so I use the external drives.

I think that the cloud system would have to be considered the mass storage and only download the data that is in use currently. That is usually a very small percentage. I would leave the mass backups and recoveries to the cloud data administrators.

 

[DaveC426913]

No, I meant what I said. Half a terabyte.

Er Nope.

 

[phinds]

I think there are some extra zeros there. Half a terabyte is only 500 GB.

Er Nope.

 

[pbuk]

500GB is not too much for cloud backup, and one of the advantages of some systems is that you can restore incrementally and selectively.

If you need to rebuild your local copy (perhaps when changing main computers), select the files you want to work on right now and they will appear almost immediately. Leave it running overnight and when you get up, all or most of it will already be done - anything you need that is not there yet you can pull forward at the click of a mouse.

Of course if your internet connection is limited you will need another solution.

 

[MikeeMiracle]

I currently have 1.2TB in cloud backup. I guess the term "backup" is a little misleading.

Traditionally when we think of backups using backup software, they backup to tape or a "backup file" which contains all of the other files rolled into 1 big file. You need the backup software that created it to extract individual files form it. While you could backup using this method and get the backup software to sync to the cloud I find it makes little sense.

A better way of looking at it is "cloud synchronisation." I used Tresorit as my cloud provider, using this software allows you specify which folders on your computer you wish to "backup" and it then synchronises the files and folders in those locations with the cloud. Whenever I make a change to a file or folder it is automatically synchronised up to the cloud within 30 seconds.

I find this "synchronisation" method more useful. I can have multiple computers connected to this service and I can choose to synchronise any/all or none of the folders with each of my devices.

I have a desktop with plenty of storage which synchronises all it's data this way. I also have a laptop with a 256GB drive. If I wish to access a file on the laptop I just download it, edit it and re-upload it again. This file is then automatically synchronised to the desktop. It's all very seamless just like having a local file server you synchronise to.

On the privacy side, the provider I use called Tresorit is very secure. All your data is encrypted locally on your device and only this encrypted data is sent to the cloud. The encryption uses your password to create a unique encryption key and only that key can be used to decrypt your data. All Tresorit can see is that I have an account and they can see the physical amount of space my data is using but that's about it. There is no "master encryption key" they an use to decrypt and view your data. This means they cannot decrypt your data even if they wanted to so its safe from even security services. The downside to using this method of local encryption is that they also have no way of recovering your data if you forget your password so the onus is very much on you to remember it.

 

[FactChecker]

I currently have 1.2TB in cloud backup. I guess the term "backup" is a little misleading.
...
A better way of looking at it is "cloud synchronisation." I used Tresorit as my cloud provider, using this software allows you specify which folders on your computer you wish to "backup" and it then synchronises the files and folders in those locations with the cloud. Whenever I make a change to a file or folder it is automatically synchronised up to the cloud within 30 seconds.

Exactly. You are using the cloud as your main mass data storage device and are completely freed from the backup responsibility and knowledge. It is really difficult to run a good backup system. That is the job of a system administrator. The cloud device holding one copy of your data might fail and be replaced without you even being aware of it.

 

[MikeeMiracle]

I am a system admin, the way I do it is more of what its called a "Business Continuity / Diaster Recovery (DR)" solution as the data is always online and live from any location.

The MASSIVE mistake people make is to confuse DR with backups and think they are on in the same thing, they are not. My DR solution does not protect me from being hit by malware which encrypts all the files as the encrypted files would get uploaded also. For that you need a backup, i.e an OFFLINE solution also, the simplest of which is an external drive you duplicate your files to.

This is what I meant by having both ONLINE and OFFLINE solutions. It's the only way to truly be protected from data loss.

Oh and periodically check your backups to make sure they are valid and restorable, there are so many stories of companies backing up data and not testing if they can actually restore it until the have to and find out their backups are useless for whatever reason.

 

[DaveC426913]

Oh and periodically check your backups to make sure they are valid and restorable

Tru dat.

 

[rcgldr]

My ISP throttles upload speed (< 20Mbps), so cloud backup isn't a reasonable option for me, at least not for the collection of videos I have. I have 2 desktops with multiple drives and each are multi-boot (Win XP, Win XP X64, Win 7 Pro 64 bit, one of them also Win 10 Pro 64 bit). I backup entire partitions to folders on other drives (while running on one of the "other" operating systems), and also to a set of external drives that I alternate through, or I do system image backup for Win 7 or Win 10 OS partition. I also backup our smart phones to the desktop drives. Occasionally, I'll swap out drives, using a disk cloning utility to duplicate the drive. I do this very carefully, to make sure I copy in the correct direction.

Some of the free cloud servers offered to me when I bought items like smart phones or a lap top have recently shut down.

I mostly use external 2TB hard drives for backup, and a 1 TB SSD drive for file transfer between systems. My concern with SSD drives is that data is lost if the internal mapping is ever messed up. However, there are large servers that are SSD only, using Raid 6 or Raid 6 variation with triple parity, but those servers will always check for unreported errors from hard drives or SSD.

 

[PAllen]

I use a very simple system which totally saved me once. First, I simply don't trust security on the cloud - it has been hacked, and I don't control it. On my windows 10 systems, I force no usage of cloud at all, for any purpose whatsoever. I simply have external drives that I store far away from their back up target, and back up both data files and system images. When a disk had a catastophic failure, I bought a replacement and restored a system image and did not have to reinstall or reconfigure a single thing. Totally up and running with no loss within a few hours. When I wanted an earlier version of a file, I just used the data file backups.

For me, additional music, photos, emails, etc. are such that I only do backup operations every few days. This of course varies by individual usage.

 

[eltodesukane]

There are 2 kinds of hard drives, those who have failed, and those who will fail.
Generally speaking, you can rely on your hard drive for 3 to 5 years on average.
So any data you want to keep must have 2 copies beside the original, and those copies must be recopied every 3 years, and the hard drives replaced as needed.
Hard drives are not a long term storage solution, unless triplicated and actively monitored, maintained and replaced.

 

[Svein]

There are 2 kinds of hard drives, those who have failed, and those who will fail.

A short answer: RAID 1 (or 5 or 10).

 

[George Keeling]

I use Google "Backup and sync" which costs £20 per year for 100GB. Whenever a file changes it gets copied up to the Google cloud and old versions of the file are kept for about a month. That is useful for accidental file wipe outs etc. When my computer died suddenly two years ago I was able to recover everything. My new computer is Windows 10. There was a problem because Windows Defender and Google Drive (Backup and Sync) seem to be incompatible! I got round this and everything is easy again. but ...

I simply don't trust security on the cloud it has been hacked

Another problem is: say a virus infected my computer and corrupted all the files for ransom and they all got into my Google cloud. To protect against this, or Pallen's hack, I have another old computer also connected to the cloud which I turn on once every week or month. The 100 or so files that have changed then get copied down to that computer. It gets turned off again. I would have to be unaware of ransom attack or hack for this system to fail. I think.

No doubt other cloud systems would do as well.

 

[Vanadium 50]

There are 2 kinds of hard drives, those who have failed, and those who will fail.

That's true for every technology. As the old joke goes "now we're just negotiating the price".

You say 3-5 years for a spinning disk. It's probably more than 10 years for a disk that you spin up only once a year. LTO is said to last 30 years. M-disc claims 1000 years, but obviously this isn't something tested. How good does it have to be? How much money are you willing to invest?

As it happens, I had a drive failure last night. Drive (a Toshiba 4 TB) and controller got into a funky state. This happens every 15000 hours or so with these drives.

Everything worked as it should. In the fortnightly consistency check (which happened at the same time as the backup - which is atypical) I got 1.11M checksum errors. So ZFS faulted the drive, degraded the array, and sent me email. I power cycled the system (the usual cure), the drive recovered, and ZFS proceeded to rebuild the array. No user intervention needed (beyond the power cycle), no loss of data, no loss in capabilities (the backup kept right on going) and a minimal loss in performance.

 

[rcgldr]

Traditionally when we think of backups using backup software, they backup to tape or a "backup file" which contains all of the other files rolled into 1 big file.

Depends on the backup software. Some tape backup apps backup folders and files. Some image backup allow for the image to be accessed as a read-only partition in order to do selective folder | file restore. I actually wrote a similar tape backup program for an Atari ST back in the 1980's (image backup, but show as partition for restore, with the FAT table cached in memory to reduce the number of random accesses).

I have a multi-boot system (XP, XP-X64, Win 7, Win 10), and wrote a folder|file copy program that include security and reparse information that I run from one instance of an operating system to backup another. It works fine for XP and XP-X64, but there's something missing for Win 7 restore. It runs, but I get some access errors running browsers, so I suspect some reparse info in Win 7 that XP isn't picking up. I have separate partitions for the OS and installed program | data, so I can use it for my Win 7 and Win 10 program | data partitions. Currently I'm using system image backups for the Win 7 | Win 10 partitions. I may buy one of the image backup programs that allow for folder | file restore.

 

[rcgldr]

I occasionally do full restores in order to "refresh" the data on my hard drives. I don't know if this would be possible or needed with solid state drives.

 

[Vanadium 50]

I occasionally do full restores in order to "refresh" the data on my hard drives

How do you know you aren't replacing good data with corrupted data?

 

[rcgldr]

I occasionally do full restores in order to "refresh" the data on my hard drives.

How do you know you aren't replacing good data with corrupted data?

I do a folder + file compare after each folder + file backup of a partition. For the image backups, I'll make two of each. I haven't tried comparing image backups yet. I may buy software that creates and checks hashes (like SHA256) keys to verify the backed up files in an image backup.

In my case, I have a multi-boot system, and run the folder + file backup from a different OS than the one being backed up. There are some standalone utilities that do something similar, but I haven't tried any of them yet.

I've also bought new hard drives to replace old ones, using standalone software to clone the drives. Just need to be very careful that you copy in the correct direction.

 

[Tom.G]

Windows at least to have two utilities for file comparison FC.EXE and WINDIFF.EXE. At least WINDIFF (a graphical interface) will also do Directory compares.

 

[DaveC426913]

I've made a start at protecting my files and my wife's files with (another) external drive - a tiny 500Gb drive the size of a short stack of credit cards. This will do for the short-term.

I will figure out some sort of auto-backup process, and I will add redundancy with cloud backup.

 

[vela]

I've made a start at protecting my files and my wife's files with (another) external drive - a tiny 500Gb drive the size of a short stack of credit cards. This will do for the short-term.

I will figure out some sort of auto-backup process, and I will add redundancy with cloud backup.

I recommend getting network attached storage for your home network and use a backup utility that can back up to it. Have this run automatically so it doesn't rely on anyone to make the time to actually make a backup as this is one of those tasks that often gets put off later and later until it's too late. I've had plenty of external drives around here for years, but my laptop rarely was backed up because I would just never get around to it.

The situation finally changed when Apple introduced Time Machine into OS X, and it could back up over the network. I used to have a drive hanging off the router to which all my home systems could back up to. I recently replaced it with a Synology Diskstation that I received for Christmas. In an expensive post-holiday inpulse-purchase buying spree, I got four 8-TB drives to install in it. They're configured as a RAID 5 (I think) so if one of the drives fails, I won't lose any data.

For online backups, I use Arq and back up to Backblaze B2 cloud storage. I have about 400 GB stored online, and it costs less than $2 a month.

I've been happy with Arq, so I'll give it a plug here. It can back up to an NAS, so it could presumably be a complete solution for you if you're running Windows or if you have a Mac. No Linux version as far as I know.

 

[PAllen]

I recommend getting network attached storage for your home network and use a backup utility that can back up to it. Have this run automatically so it doesn't rely on anyone to make the time to actually make a backup as this is one of those tasks that often gets put off later and later until it's too late. I've had plenty of external drives around here for years, but my laptop rarely was backed up because I would just never get around to it.

The situation finally changed when Apple introduced Time Machine into OS X, and it could back up over the network. I used to have a drive hanging off the router to which all my home systems could back up to. I recently replaced it with a Synology Diskstation that I received for Christmas. In an expensive post-holiday inpulse-purchase buying spree, I got four 8-TB drives to install in it. They're configured as a RAID 5 (I think) so if one of the drives fails, I won't lose any data.

For online backups, I use Arq and back up to Backblaze B2 cloud storage. I have about 400 GB stored online, and it costs less than $2 a month.

I've been happy with Arq, so I'll give it a plug here. It can back up to an NAS, so it could presumably be a complete solution for you if you're running Windows or if you have a Mac. No Linux version as far as I know.

The problem I have with any such backup scheme is that it relies on always connected devices. One lightning strike to a nearby tree could fry all attached devices ( exactly this happened to my father). I do not believe surge protectors are sufficient to prevent this in an extreme case ( I have them anyway).Thus, with simpler tools, but greater redundancy, I rely on drives connected only while in process of backing up, stored disconnected from power at multiple locations. I may occasionally go 10 days without backing up if I am distracted by life, but losing a week of stuff is much better than losing everything.

Cloud I never use and do not trust for anything at all.

 

[dlgoff]

... an old DEC rack in the garage
View attachment 278227 ...

Nice garage and rack.

 

[Vanadium 50]

Nice garage and rack.

When I was young, "nice rack" meant something else. But I digress.

You have a lot of advice here, but I think you need to clarify what "a certain amount of rot" means. For $0, you will get a total wipe-out every 5 years or so. For $1,000,000, your data will last for a century or more. Somewhere in between is where you want to be.

If you tell us where, we can help you get there.

 

[sbrothy]

Summary:: I want to stop losing my precious projects. I need a reliable way to backup files for me, and friends & fam

I thought I had a long-term backup of my important stuff when I bought a 1Tb Seagate External Drive. It died and took years of projects and memories with it.

Running Win10 on a laptop.

I think there are three options:

1. Another (better) external drive
2. Some newfangled solid-state external drive
3. Cloud storage

Some considerations:

• I want to ensure it covers mine and my wife's files - (and, in an ideal world, daughter-in-law-at-separate-residence). Whether that means one, two or three devices/services.
• I think I'll need to set up an auto-backup service for my wife. I personally didn't need one, but I'm more conscientious about securing my important files. But maybe I'll let a computer do something for me that I can do myself. (Big step there.)
• I don't want to spend a half hour every month watching progress bars push my files around. (I guess that's auto backup during off hours)
• I do have my own webhost and domain, if that makes any diff. There is storage there. I'm just not sure it's a serious, robust consideration (I've had bad experiences with webhosts).

Anyway, I'm looking for advice and personal opinions about what to go with.

This may be a little overkill but if you're considering RAIDing maybe not:

https://hackaday.io/project/175094-raspberry-pi-ceph-cluster
https://docs.ceph.com/en/latest/ceph-volume/drive-group/

 

[Vanadium 50]

Ceph is not exactly turn-key. It's not clear to me that having two drives Cephed is any better than having two USB drives and writing it twice.

 

[vela]

The problem I have with any such backup scheme is that it relies on always connected devices. One lightning strike to a nearby tree could fry all attached devices ( exactly this happened to my father). I do not believe surge protectors are sufficient to prevent this in an extreme case ( I have them anyway). Thus, with simpler tools, but greater redundancy, I rely on drives connected only while in process of backing up, stored disconnected from power at multiple locations. I may occasionally go 10 days without backing up if I am distracted by life, but losing a week of stuff is much better than losing everything.

Your method works if you have the discipline to back up regularly, but many people, like me, don't. And the fact that traditionally most users didn't back up regularly even when they knew they should suggests that I'm not in the minority. Plus, I live in an area which doesn't get many lightning strikes, so that's not really a concern to me.

When I've accessed my backups, it's usually because I realized I still wanted a file I deleted a short while ago. With hourly backups, most of the time I can find what I need. On the other hand, if my last back up was a week ago, I would've been out of luck.

It also doesn't have to be an either-or situation. You could use both methods of systems simultaneously.

Cloud I never use and do not trust for anything at all.

I hear this sentiment a lot. What's the reason for your distrust?

 

[DaveC426913]

You have a lot of advice here, but I think you need to clarify what "a certain amount of rot" means. For $0, you will get a total wipe-out every 5 years or so. For $1,000,000, your data will last for a century or more. Somewhere in between is where you want to be.

If you tell us where, we can help you get there.

Exactly my thoughts. I thought I had made it clear. I am not looking for a business-grade solution, just a home-grade solution. So, no RAIDS, etc.

I'll make one-time purchases of maybe a couple hundred $$. I dislike online subscriptions because I find they rot after a few years (What's my #$%$@ username!), but that's a necessary evil.

I bought a tiny $150 500G backup drive, but I don't yet have an automated backup utility for it. And it's already inadequate - a single backup of my projects ate up half the drive space - and I haven't even backed up my wife's files yet.

 

[Vanadium 50]

Well, a RAID at home is not crazy. I have three, and while I am crazy, I'm not that crazy. Buy Windows itself will do automatic backups at a certain time. There is also commercial software for a few $10's. Your biggest risk will likely be a failing USB disk that goes unnoticed followed by a failed laptop disk.