Is file size on a 5TB drive standard for drives over 4TB?

[phinds]

TL;DR

Shocked that a 1K file takes 1Meg of disk space on my new 5TB drive

I used to know all about the FAT file structure and sectors and clusters and stuff but have forgotten it all. I was shocked to find that the new 5TB external HD I bought takes 1Meg to store a 1Kb file (and any file less than 1Meg).

Is this standard on drives over 4TB ?

Is there some other more reasonable size that some huge drives have or do I just have to live w/ this?

Why I care: It's a major bummer because I did a full back up of 303GB (326GB on disk) of mixed-sized files and it took 893Gigs on the disk (also 5 hours, but I didn't mind that). As a consequence, I'll be able to do only 1/3rd as many such archived backups as I had expected.

 

[jedishrfu]

I think this is a problem with FAT filesystems that are compatible with Windows. It has to do with a size limit for the table holding seek addresses on to the disk.

 

[phinds]

I think this is a problem with FAT filesystems that are compatible with Windows. It has to do with a size limit for the table holding seek addresses on to the disk.

So it's likely that ALL 5TB drives have this issue? I thought the FAT was long ago abandoned in favor of the NTFS.

 

[PeterDonis]

I thought the FAT was long ago abandoned in favor of the NTFS.

Windows machines have defaulted to NTFS since Windows XP, IIRC (and the "server" and "enterprise" versions of Windows, as opposed to the "home" versions, defaulted to NTFS since well before that). So I don't think the issue you're seeing is due to the filesystem.

How are you determining the space taken up on disk?

 

[jedishrfu]

For external drives they may be preformatted with one or the other based on use on windows or macos.

Macos can read but not write ntfs without a third party extension. At least it used to be that way not sure now with how everything is changing.

 

[phinds]

I'm using Windows file explorer to tell me the size. It's reliable.

I just found out that the drive uses exFAT and according to Wikipedia, it should be using TWO megs for the cluster size (HORRORS !).

So it looks as though I'll just have to live with this.

 

[phinds]

I don't suppose partitioning it would be any help?

 

[jedishrfu]

Heres a discussion on it

https://superuser.com/questions/365...ace-on-an-exfat-formatted-external-hard-drive

You might need to reformat it though.

 

[phinds]

Interesting. The article (thanks by the way) points to THIS table, which says the cluster size should be 128K, not 1Meg

 

[Mark44]

Is it possible that you could reformat the external drive to NTFS?

 

[pbuk]

1. You shouldn't be using exFAT on a backup drive, you should use a journalling file system which on a Windows machine means NTFS.
2. Is this a 'big name' product (WD, Seagate etc.)?
3. Did you format it yourself or did it come formatted?
4. Is it connected via USB?

 

[Mark44]

So it's likely that ALL 5TB drives have this issue? I thought the FAT was long ago abandoned in favor of the NTFS.

On my two desktop computers I have a 4TB external drive on one, and a 1TB ext. drive on the other. Both are NTFS.
I have a bunch of small thumb drives. It seems that most of these are FAT-formatted.

 

[phinds]

Is it possible that you could reformat the external drive to NTFS?

I'm hesitant 'cause I'm not clear on what the ramifications would be. Since the article say those are the DEFAULT size, perhaps I can reformat it w/ exFAT but using a smaller allocation unit.

 

[pbuk]

The ramifications would include not corrupting your data due to an unscheduled disconnect/power loss. Use NTFS.

 

[phinds]

1. You shouldn't be using exFAT on a backup drive, you should use a journalling file system which on a Windows machine means NTFS.

1. The Wikipedia tables I looked at seem to imply that NTFS can't handle really big drives. Is that wrong?

   [*]Is this a 'big name' product (WD, Seagate etc.)?

   No it's an off brand from Amazon but it did get really good reviews.

   [*]Did you format it yourself or did it come formatted?

   I came formatted. What do you think of the idea of reformatting w/ exFAT but a smaller allocation size?

   [*]Is it connected via USB?

   Yes, USB.

 

[phinds]

Which is why I am surprised that you have a 5TB drive formatted with exFAT in the first place, let alone with a stupid cluster size, and why I ask the other questions.

OK, thanks.

 

[phinds]

After looking into it a bit more it sure seems you were right @pbuk (yeah, you already knew that didn't you ) Looks like reformatting w/ NTFS is the way to go. Thanks.

@jedishrfu and @Mark44 my thanks to you guys as well for your help.

 

[pbuk]

The Wikipedia tables I looked at seem to imply that NTFS can't handle really big drives. Is that wrong?

It is wrong. NTFS can handle up to 8 Petabytes with a 2MB cluster, with the default 4Kb cluster it can handle up to 16TB.

No it's an off brand from Amazon but it did get really good reviews.

That may be a 'saving' you end up regretting I am afraid.

I came formatted. What do you think of the idea of reformatting w/ exFAT but a smaller allocation size?

If you don't want to lose your data you shouldn't use exFAT.

 

[phinds]

Is it possible that you could reformat the external drive to NTFS?

Turns out this is the best way to go. I just did it and I've restarted the backup and I can see that it's working great. I set the allocation size to 8Kb and the overhead w/ that is fine by me and it'll save a trivial amount of time on the larger files (as opposed to a smaller allocation size).

 

[phinds]

That may be a 'saving' you end up regretting I am afraid.

Actually, my decision was not base on price and I saved almost nothing. It was based on ratings. I always look for stuff that has a lot of ratings and a very high rating value. This drive was 4.8 and higher than the brand names.

OOPS. Turns out it IS a brand name (Western Digital) and a bit on the expensive side (not by much) but as I said, name brand is less important to me than other user's experiences with whatever I'm buying.
https://www.amazon.com/gp/product/B07VNTFHD5/?tag=pfamazon01-20

 

[jedishrfu]

The wasted space on drives has always been there. Its the nature of balancing speed and accessibility with disk storage.

One trick early programmers did was to zip files for compression and because numerous small files would be replaced by one larger file but without that wasted space.

@pbuk and @Mark44 recommendations are sound for Windows but MacOS is a different beast and NTFS is a proprietary format.

https://en.wikipedia.org/wiki/NTFS

Journaling is important everywhere from filesystems to databases. Crap happens and a corrupted directory table can really make your day. Its why some OSes would run fdisk when it detected a disk issue. I recall my old Linux boxes doing that on a bad shutdown.

https://en.wikipedia.org/wiki/Fdisk

 

[pbuk]

OOPS. Turns out it IS a brand name (Western Digital) ...
https://www.amazon.com/gp/product/B07VNTFHD5/?tag=pfamazon01-20

That explains the exFAT formatting and the large cluster size - this drive is supplied optimised for use on a PlayStation which (i) can't read NTFS and (ii) stores its data in huge files where a large cluster size adds speed with negligible wasted space.

 

[jedishrfu]

As part of our PF Outreach program, here's a HowStuffWorks page on hard drives for @phinds:

https://computer.howstuffworks.com/hard-disk.htm

 

[pbuk]

@pbuk and @Mark44 recommendations are sound for Windows but MacOS is a different beast and NTFS is a proprietary format.

Yes indeed; I checked the OP's use case. Big brand suppliers provide a driver for iOS for their NTFS backup drives, or you can buy a Mac-specific product (if you were daft enough to spend a fortune on a piece of shiny white plastic once, you may as well do it again). If you are keen enough to have a Linux system that needs backing up then you are probably rsync'ing to a EXT-4 drive on an NFS.

Journaling is important everywhere from filesystems to databases.

Yes, although without file-system journalling as well the database journal file may easily be corrupted.