Modern file backup options: cloud? / HD?

Join the discussion
Registration is free. Ask a follow-up in this thread, or start your own.
Sign up Log in
69 replies · 7K views
Physics news on Phys.org
phinds said:
I back up about about 500,000 gigs so even if it were free storage the download for a backup would be a killer.

I've written a "delta site mirror" program for my wood ID site (200,000+ JPG and HTML files) that I could expand to my whole backup but I would not be happy w/ the amount of time it would take so I use the external drives.
I think that the cloud system would have to be considered the mass storage and only download the data that is in use currently. That is usually a very small percentage. I would leave the mass backups and recoveries to the cloud data administrators.
 
  • Like
Likes   Reactions: pbuk
phinds said:
No, I meant what I said. Half a terabyte.
Er Nope. :wink:
 
Vanadium 50 said:
I think there are some extra zeros there. Half a terabyte is only 500 GB.
DaveC426913 said:
Er Nope. :wink:
 

Attachments

  • egg_small.jpg
    egg_small.jpg
    1.5 KB · Views: 190
  • Like
Likes   Reactions: DaveC426913, FactChecker and Vanadium 50
500GB is not too much for cloud backup, and one of the advantages of some systems is that you can restore incrementally and selectively.

If you need to rebuild your local copy (perhaps when changing main computers), select the files you want to work on right now and they will appear almost immediately. Leave it running overnight and when you get up, all or most of it will already be done - anything you need that is not there yet you can pull forward at the click of a mouse.

Of course if your internet connection is limited you will need another solution.
 
I currently have 1.2TB in cloud backup. I guess the term "backup" is a little misleading.

Traditionally when we think of backups using backup software, they backup to tape or a "backup file" which contains all of the other files rolled into 1 big file. You need the backup software that created it to extract individual files form it. While you could backup using this method and get the backup software to sync to the cloud I find it makes little sense.

A better way of looking at it is "cloud synchronisation." I used Tresorit as my cloud provider, using this software allows you specify which folders on your computer you wish to "backup" and it then synchronises the files and folders in those locations with the cloud. Whenever I make a change to a file or folder it is automatically synchronised up to the cloud within 30 seconds.

I find this "synchronisation" method more useful. I can have multiple computers connected to this service and I can choose to synchronise any/all or none of the folders with each of my devices.

I have a desktop with plenty of storage which synchronises all it's data this way. I also have a laptop with a 256GB drive. If I wish to access a file on the laptop I just download it, edit it and re-upload it again. This file is then automatically synchronised to the desktop. It's all very seamless just like having a local file server you synchronise to.

On the privacy side, the provider I use called Tresorit is very secure. All your data is encrypted locally on your device and only this encrypted data is sent to the cloud. The encryption uses your password to create a unique encryption key and only that key can be used to decrypt your data. All Tresorit can see is that I have an account and they can see the physical amount of space my data is using but that's about it. There is no "master encryption key" they an use to decrypt and view your data. This means they cannot decrypt your data even if they wanted to so its safe from even security services. The downside to using this method of local encryption is that they also have no way of recovering your data if you forget your password so the onus is very much on you to remember it.
 
  • Like
Likes   Reactions: FactChecker and pbuk
MikeeMiracle said:
I currently have 1.2TB in cloud backup. I guess the term "backup" is a little misleading.
...
A better way of looking at it is "cloud synchronisation." I used Tresorit as my cloud provider, using this software allows you specify which folders on your computer you wish to "backup" and it then synchronises the files and folders in those locations with the cloud. Whenever I make a change to a file or folder it is automatically synchronised up to the cloud within 30 seconds.
Exactly. You are using the cloud as your main mass data storage device and are completely freed from the backup responsibility and knowledge. It is really difficult to run a good backup system. That is the job of a system administrator. The cloud device holding one copy of your data might fail and be replaced without you even being aware of it.
 
Last edited:
I am a system admin, the way I do it is more of what its called a "Business Continuity / Diaster Recovery (DR)" solution as the data is always online and live from any location.

The MASSIVE mistake people make is to confuse DR with backups and think they are on in the same thing, they are not. My DR solution does not protect me from being hit by malware which encrypts all the files as the encrypted files would get uploaded also. For that you need a backup, i.e an OFFLINE solution also, the simplest of which is an external drive you duplicate your files to.

This is what I meant by having both ONLINE and OFFLINE solutions. It's the only way to truly be protected from data loss.

Oh and periodically check your backups to make sure they are valid and restorable, there are so many stories of companies backing up data and not testing if they can actually restore it until the have to and find out their backups are useless for whatever reason.
 
  • Like
Likes   Reactions: FactChecker and nsaspook
MikeeMiracle said:
Oh and periodically check your backups to make sure they are valid and restorable
Tru dat.
 
  • Like
Likes   Reactions: FactChecker
My ISP throttles upload speed (< 20Mbps), so cloud backup isn't a reasonable option for me, at least not for the collection of videos I have. I have 2 desktops with multiple drives and each are multi-boot (Win XP, Win XP X64, Win 7 Pro 64 bit, one of them also Win 10 Pro 64 bit). I backup entire partitions to folders on other drives (while running on one of the "other" operating systems), and also to a set of external drives that I alternate through, or I do system image backup for Win 7 or Win 10 OS partition. I also backup our smart phones to the desktop drives. Occasionally, I'll swap out drives, using a disk cloning utility to duplicate the drive. I do this very carefully, to make sure I copy in the correct direction.

Some of the free cloud servers offered to me when I bought items like smart phones or a lap top have recently shut down.

I mostly use external 2TB hard drives for backup, and a 1 TB SSD drive for file transfer between systems. My concern with SSD drives is that data is lost if the internal mapping is ever messed up. However, there are large servers that are SSD only, using Raid 6 or Raid 6 variation with triple parity, but those servers will always check for unreported errors from hard drives or SSD.
 
Last edited:
I use a very simple system which totally saved me once. First, I simply don't trust security on the cloud - it has been hacked, and I don't control it. On my windows 10 systems, I force no usage of cloud at all, for any purpose whatsoever. I simply have external drives that I store far away from their back up target, and back up both data files and system images. When a disk had a catastophic failure, I bought a replacement and restored a system image and did not have to reinstall or reconfigure a single thing. Totally up and running with no loss within a few hours. When I wanted an earlier version of a file, I just used the data file backups.

For me, additional music, photos, emails, etc. are such that I only do backup operations every few days. This of course varies by individual usage.
 
There are 2 kinds of hard drives, those who have failed, and those who will fail.
Generally speaking, you can rely on your hard drive for 3 to 5 years on average.
So any data you want to keep must have 2 copies beside the original, and those copies must be recopied every 3 years, and the hard drives replaced as needed.
Hard drives are not a long term storage solution, unless triplicated and actively monitored, maintained and replaced.
 
eltodesukane said:
There are 2 kinds of hard drives, those who have failed, and those who will fail.
A short answer: RAID 1 (or 5 or 10).
 
  • Like
Likes   Reactions: Vanadium 50 and FactChecker
I use Google "Backup and sync" which costs £20 per year for 100GB. Whenever a file changes it gets copied up to the Google cloud and old versions of the file are kept for about a month. That is useful for accidental file wipe outs etc. When my computer died suddenly two years ago I was able to recover everything. My new computer is Windows 10. There was a problem because Windows Defender and Google Drive (Backup and Sync) seem to be incompatible! I got round this and everything is easy again. but ...
PAllen said:
I simply don't trust security on the cloud it has been hacked
Another problem is: say a virus infected my computer and corrupted all the files for ransom and they all got into my Google cloud. To protect against this, or Pallen's hack, I have another old computer also connected to the cloud which I turn on once every week or month. The 100 or so files that have changed then get copied down to that computer. It gets turned off again. I would have to be unaware of ransom attack or hack for this system to fail. I think.

No doubt other cloud systems would do as well.
 
eltodesukane said:
There are 2 kinds of hard drives, those who have failed, and those who will fail.

That's true for every technology. As the old joke goes "now we're just negotiating the price".

You say 3-5 years for a spinning disk. It's probably more than 10 years for a disk that you spin up only once a year. LTO is said to last 30 years. M-disc claims 1000 years, but obviously this isn't something tested. How good does it have to be? How much money are you willing to invest?

As it happens, I had a drive failure last night. Drive (a Toshiba 4 TB) and controller got into a funky state. This happens every 15000 hours or so with these drives.

Everything worked as it should. In the fortnightly consistency check (which happened at the same time as the backup - which is atypical) I got 1.11M checksum errors. So ZFS faulted the drive, degraded the array, and sent me email. I power cycled the system (the usual cure), the drive recovered, and ZFS proceeded to rebuild the array. No user intervention needed (beyond the power cycle), no loss of data, no loss in capabilities (the backup kept right on going) and a minimal loss in performance.
 
MikeeMiracle said:
Traditionally when we think of backups using backup software, they backup to tape or a "backup file" which contains all of the other files rolled into 1 big file.
Depends on the backup software. Some tape backup apps backup folders and files. Some image backup allow for the image to be accessed as a read-only partition in order to do selective folder | file restore. I actually wrote a similar tape backup program for an Atari ST back in the 1980's (image backup, but show as partition for restore, with the FAT table cached in memory to reduce the number of random accesses).

I have a multi-boot system (XP, XP-X64, Win 7, Win 10), and wrote a folder|file copy program that include security and reparse information that I run from one instance of an operating system to backup another. It works fine for XP and XP-X64, but there's something missing for Win 7 restore. It runs, but I get some access errors running browsers, so I suspect some reparse info in Win 7 that XP isn't picking up. I have separate partitions for the OS and installed program | data, so I can use it for my Win 7 and Win 10 program | data partitions. Currently I'm using system image backups for the Win 7 | Win 10 partitions. I may buy one of the image backup programs that allow for folder | file restore.
 
I occasionally do full restores in order to "refresh" the data on my hard drives. I don't know if this would be possible or needed with solid state drives.
 
rcgldr said:
I occasionally do full restores in order to "refresh" the data on my hard drives.

Vanadium 50 said:
How do you know you aren't replacing good data with corrupted data?
I do a folder + file compare after each folder + file backup of a partition. For the image backups, I'll make two of each. I haven't tried comparing image backups yet. I may buy software that creates and checks hashes (like SHA256) keys to verify the backed up files in an image backup.

In my case, I have a multi-boot system, and run the folder + file backup from a different OS than the one being backed up. There are some standalone utilities that do something similar, but I haven't tried any of them yet.

I've also bought new hard drives to replace old ones, using standalone software to clone the drives. Just need to be very careful that you copy in the correct direction.
 
Last edited:
  • Like
Likes   Reactions: Vanadium 50
Windows at least to have two utilities for file comparison FC.EXE and WINDIFF.EXE. At least WINDIFF (a graphical interface) will also do Directory compares.
 
I've made a start at protecting my files and my wife's files with (another) external drive - a tiny 500Gb drive the size of a short stack of credit cards. This will do for the short-term.

I will figure out some sort of auto-backup process, and I will add redundancy with cloud backup.
 
DaveC426913 said:
I've made a start at protecting my files and my wife's files with (another) external drive - a tiny 500Gb drive the size of a short stack of credit cards. This will do for the short-term.

I will figure out some sort of auto-backup process, and I will add redundancy with cloud backup.
I recommend getting network attached storage for your home network and use a backup utility that can back up to it. Have this run automatically so it doesn't rely on anyone to make the time to actually make a backup as this is one of those tasks that often gets put off later and later until it's too late. I've had plenty of external drives around here for years, but my laptop rarely was backed up because I would just never get around to it.

The situation finally changed when Apple introduced Time Machine into OS X, and it could back up over the network. I used to have a drive hanging off the router to which all my home systems could back up to. I recently replaced it with a Synology Diskstation that I received for Christmas. In an expensive post-holiday inpulse-purchase buying spree, I got four 8-TB drives to install in it. They're configured as a RAID 5 (I think) so if one of the drives fails, I won't lose any data.

For online backups, I use Arq and back up to Backblaze B2 cloud storage. I have about 400 GB stored online, and it costs less than $2 a month.

I've been happy with Arq, so I'll give it a plug here. It can back up to an NAS, so it could presumably be a complete solution for you if you're running Windows or if you have a Mac. No Linux version as far as I know.
 
  • Like
Likes   Reactions: DaveC426913
vela said:
I recommend getting network attached storage for your home network and use a backup utility that can back up to it. Have this run automatically so it doesn't rely on anyone to make the time to actually make a backup as this is one of those tasks that often gets put off later and later until it's too late. I've had plenty of external drives around here for years, but my laptop rarely was backed up because I would just never get around to it.

The situation finally changed when Apple introduced Time Machine into OS X, and it could back up over the network. I used to have a drive hanging off the router to which all my home systems could back up to. I recently replaced it with a Synology Diskstation that I received for Christmas. In an expensive post-holiday inpulse-purchase buying spree, I got four 8-TB drives to install in it. They're configured as a RAID 5 (I think) so if one of the drives fails, I won't lose any data.

For online backups, I use Arq and back up to Backblaze B2 cloud storage. I have about 400 GB stored online, and it costs less than $2 a month.

I've been happy with Arq, so I'll give it a plug here. It can back up to an NAS, so it could presumably be a complete solution for you if you're running Windows or if you have a Mac. No Linux version as far as I know.
The problem I have with any such backup scheme is that it relies on always connected devices. One lightning strike to a nearby tree could fry all attached devices ( exactly this happened to my father). I do not believe surge protectors are sufficient to prevent this in an extreme case ( I have them anyway).Thus, with simpler tools, but greater redundancy, I rely on drives connected only while in process of backing up, stored disconnected from power at multiple locations. I may occasionally go 10 days without backing up if I am distracted by life, but losing a week of stuff is much better than losing everything.

Cloud I never use and do not trust for anything at all.
 
dlgoff said:
Nice garage and rack.

When I was young, "nice rack" meant something else. But I digress.

You have a lot of advice here, but I think you need to clarify what "a certain amount of rot" means. For $0, you will get a total wipe-out every 5 years or so. For $1,000,000, your data will last for a century or more. Somewhere in between is where you want to be.

If you tell us where, we can help you get there.
 
DaveC426913 said:
Summary:: I want to stop losing my precious projects. I need a reliable way to backup files for me, and friends & fam

I thought I had a long-term backup of my important stuff when I bought a 1Tb Seagate External Drive. It died and took years of projects and memories with it.

Running Win10 on a laptop.

I think there are three options:
  1. Another (better) external drive
  2. Some newfangled solid-state external drive
  3. Cloud storage
Some considerations:
  • I want to ensure it covers mine and my wife's files - (and, in an ideal world, daughter-in-law-at-separate-residence). Whether that means one, two or three devices/services.
  • I think I'll need to set up an auto-backup service for my wife. I personally didn't need one, but I'm more conscientious about securing my important files. But maybe I'll let a computer do something for me that I can do myself. (Big step there.)
  • I don't want to spend a half hour every month watching progress bars push my files around. (I guess that's auto backup during off hours)
  • I do have my own webhost and domain, if that makes any diff. There is storage there. I'm just not sure it's a serious, robust consideration (I've had bad experiences with webhosts).
Anyway, I'm looking for advice and personal opinions about what to go with.

This may be a little overkill but if you're considering RAIDing maybe not:

https://hackaday.io/project/175094-raspberry-pi-ceph-cluster
https://docs.ceph.com/en/latest/ceph-volume/drive-group/
 
PAllen said:
The problem I have with any such backup scheme is that it relies on always connected devices. One lightning strike to a nearby tree could fry all attached devices ( exactly this happened to my father). I do not believe surge protectors are sufficient to prevent this in an extreme case ( I have them anyway). Thus, with simpler tools, but greater redundancy, I rely on drives connected only while in process of backing up, stored disconnected from power at multiple locations. I may occasionally go 10 days without backing up if I am distracted by life, but losing a week of stuff is much better than losing everything.
Your method works if you have the discipline to back up regularly, but many people, like me, don't. And the fact that traditionally most users didn't back up regularly even when they knew they should suggests that I'm not in the minority. Plus, I live in an area which doesn't get many lightning strikes, so that's not really a concern to me.

When I've accessed my backups, it's usually because I realized I still wanted a file I deleted a short while ago. With hourly backups, most of the time I can find what I need. On the other hand, if my last back up was a week ago, I would've been out of luck.

It also doesn't have to be an either-or situation. You could use both methods of systems simultaneously.

Cloud I never use and do not trust for anything at all.
I hear this sentiment a lot. What's the reason for your distrust?
 
Vanadium 50 said:
You have a lot of advice here, but I think you need to clarify what "a certain amount of rot" means. For $0, you will get a total wipe-out every 5 years or so. For $1,000,000, your data will last for a century or more. Somewhere in between is where you want to be.

If you tell us where, we can help you get there.
Exactly my thoughts. I thought I had made it clear. I am not looking for a business-grade solution, just a home-grade solution. So, no RAIDS, etc.

I'll make one-time purchases of maybe a couple hundred $$. I dislike online subscriptions because I find they rot after a few years (What's my #$%$@ username!), but that's a necessary evil.

I bought a tiny $150 500G backup drive, but I don't yet have an automated backup utility for it. And it's already inadequate - a single backup of my projects ate up half the drive space - and I haven't even backed up my wife's files yet.
 
Well, a RAID at home is not crazy. I have three, and while I am crazy, I'm not that crazy. Buy Windows itself will do automatic backups at a certain time. There is also commercial software for a few $10's. Your biggest risk will likely be a failing USB disk that goes unnoticed followed by a failed laptop disk.
 
  • Like
Likes   Reactions: vela and russ_watters