Is file size on a 5TB drive standard for drives over 4TB?

In summary: It's an off brand from Amazon but it did get really good reviews.It seems that reformatting w/ exFAT but a smaller allocation size might be an option.Reformatting with NTFS is the way to go. Thanks.
  • #1
phinds
Science Advisor
Insights Author
Gold Member
18,707
13,410
TL;DR Summary
Shocked that a 1K file takes 1Meg of disk space on my new 5TB drive
I used to know all about the FAT file structure and sectors and clusters and stuff but have forgotten it all. I was shocked to find that the new 5TB external HD I bought takes 1Meg to store a 1Kb file (and any file less than 1Meg).

Is this standard on drives over 4TB ?

Is there some other more reasonable size that some huge drives have or do I just have to live w/ this?

Why I care: It's a major bummer because I did a full back up of 303GB (326GB on disk) of mixed-sized files and it took 893Gigs on the disk (also 5 hours, but I didn't mind that). As a consequence, I'll be able to do only 1/3rd as many such archived backups as I had expected.
 
Computer science news on Phys.org
  • #2
I think this is a problem with FAT filesystems that are compatible with Windows. It has to do with a size limit for the table holding seek addresses on to the disk.
 
  • Like
Likes phinds
  • #3
jedishrfu said:
I think this is a problem with FAT filesystems that are compatible with Windows. It has to do with a size limit for the table holding seek addresses on to the disk.
So it's likely that ALL 5TB drives have this issue? I thought the FAT was long ago abandoned in favor of the NTFS.
 
  • #4
phinds said:
I thought the FAT was long ago abandoned in favor of the NTFS.

Windows machines have defaulted to NTFS since Windows XP, IIRC (and the "server" and "enterprise" versions of Windows, as opposed to the "home" versions, defaulted to NTFS since well before that). So I don't think the issue you're seeing is due to the filesystem.

How are you determining the space taken up on disk?
 
  • #5
For external drives they may be preformatted with one or the other based on use on windows or macos.

Macos can read but not write ntfs without a third party extension. At least it used to be that way not sure now with how everything is changing.
 
  • #6
I'm using Windows file explorer to tell me the size. It's reliable.

I just found out that the drive uses exFAT and according to Wikipedia, it should be using TWO megs for the cluster size (HORRORS !).
1600284665752.png


So it looks as though I'll just have to live with this.
 
  • #7
I don't suppose partitioning it would be any help?
 
  • #9
Interesting. The article (thanks by the way) points to THIS table, which says the cluster size should be 128K, not 1Meg
1600285384738.png
 
  • #10
Is it possible that you could reformat the external drive to NTFS?
 
  • #11
  1. You shouldn't be using exFAT on a backup drive, you should use a journalling file system which on a Windows machine means NTFS.
  2. Is this a 'big name' product (WD, Seagate etc.)?
  3. Did you format it yourself or did it come formatted?
  4. Is it connected via USB?
 
  • #12
phinds said:
So it's likely that ALL 5TB drives have this issue? I thought the FAT was long ago abandoned in favor of the NTFS.
On my two desktop computers I have a 4TB external drive on one, and a 1TB ext. drive on the other. Both are NTFS.
I have a bunch of small thumb drives. It seems that most of these are FAT-formatted.
 
  • #13
Mark44 said:
Is it possible that you could reformat the external drive to NTFS?
I'm hesitant 'cause I'm not clear on what the ramifications would be. Since the article say those are the DEFAULT size, perhaps I can reformat it w/ exFAT but using a smaller allocation unit.
 
  • #14
The ramifications would include not corrupting your data due to an unscheduled disconnect/power loss. Use NTFS.
 
  • Like
Likes jedishrfu and etotheipi
  • #15
pbuk said:
  1. You shouldn't be using exFAT on a backup drive, you should use a journalling file system which on a Windows machine means NTFS.
  1. The Wikipedia tables I looked at seem to imply that NTFS can't handle really big drives. Is that wrong?
    [*]Is this a 'big name' product (WD, Seagate etc.)?
    No it's an off brand from Amazon but it did get really good reviews.
    [*]Did you format it yourself or did it come formatted?
    I came formatted. What do you think of the idea of reformatting w/ exFAT but a smaller allocation size?
    [*]Is it connected via USB?
    Yes, USB.
 
  • #16
pbuk said:
Which is why I am surprised that you have a 5TB drive formatted with exFAT in the first place, let alone with a stupid cluster size, and why I ask the other questions.
OK, thanks.
 
  • #17
After looking into it a bit more it sure seems you were right @pbuk (yeah, you already knew that didn't you :oldlaugh: ) Looks like reformatting w/ NTFS is the way to go. Thanks.

@jedishrfu and @Mark44 my thanks to you guys as well for your help.
 
  • Like
Likes pbuk
  • #18
phinds said:
The Wikipedia tables I looked at seem to imply that NTFS can't handle really big drives. Is that wrong?
It is wrong. NTFS can handle up to 8 Petabytes with a 2MB cluster, with the default 4Kb cluster it can handle up to 16TB.
phinds said:
No it's an off brand from Amazon but it did get really good reviews.
That may be a 'saving' you end up regretting I am afraid.
phinds said:
I came formatted. What do you think of the idea of reformatting w/ exFAT but a smaller allocation size?
If you don't want to lose your data you shouldn't use exFAT.
 
  • Informative
Likes phinds
  • #19
Mark44 said:
Is it possible that you could reformat the external drive to NTFS?
Turns out this is the best way to go. I just did it and I've restarted the backup and I can see that it's working great. I set the allocation size to 8Kb and the overhead w/ that is fine by me and it'll save a trivial amount of time on the larger files (as opposed to a smaller allocation size).
 
  • Like
  • Informative
Likes anorlunda and pbuk
  • #20
pbuk said:
That may be a 'saving' you end up regretting I am afraid.
Actually, my decision was not base on price and I saved almost nothing. It was based on ratings. I always look for stuff that has a lot of ratings and a very high rating value. This drive was 4.8 and higher than the brand names.

OOPS. Turns out it IS a brand name (Western Digital) and a bit on the expensive side (not by much) but as I said, name brand is less important to me than other user's experiences with whatever I'm buying.
https://www.amazon.com/gp/product/B07VNTFHD5/?tag=pfamazon01-20
 
  • #21
The wasted space on drives has always been there. Its the nature of balancing speed and accessibility with disk storage.

One trick early programmers did was to zip files for compression and because numerous small files would be replaced by one larger file but without that wasted space.

@pbuk and @Mark44 recommendations are sound for Windows but MacOS is a different beast and NTFS is a proprietary format.

https://en.wikipedia.org/wiki/NTFS

Journaling is important everywhere from filesystems to databases. Crap happens and a corrupted directory table can really make your day. Its why some OSes would run fdisk when it detected a disk issue. I recall my old Linux boxes doing that on a bad shutdown.

https://en.wikipedia.org/wiki/Fdisk
 
  • Like
Likes pbuk
  • #22
phinds said:
OOPS. Turns out it IS a brand name (Western Digital) ...
https://www.amazon.com/gp/product/B07VNTFHD5/?tag=pfamazon01-20
That explains the exFAT formatting and the large cluster size - this drive is supplied optimised for use on a PlayStation which (i) can't read NTFS and (ii) stores its data in huge files where a large cluster size adds speed with negligible wasted space.
 
Last edited:
  • Informative
  • Like
Likes phinds and jedishrfu
  • #24
jedishrfu said:
@pbuk and @Mark44 recommendations are sound for Windows but MacOS is a different beast and NTFS is a proprietary format.
Yes indeed; I checked the OP's use case. Big brand suppliers provide a driver for iOS for their NTFS backup drives, or you can buy a Mac-specific product (if you were daft enough to spend a fortune on a piece of shiny white plastic once, you may as well do it again). If you are keen enough to have a Linux system that needs backing up then you are probably rsync'ing to a EXT-4 drive on an NFS.

jedishrfu said:
Journaling is important everywhere from filesystems to databases.
Yes, although without file-system journalling as well the database journal file may easily be corrupted.
 
  • Like
Likes jedishrfu

1. What is the maximum file size that can be stored on a 5TB drive?

The maximum file size that can be stored on a 5TB drive is 5 terabytes. This is because the drive has a total storage capacity of 5TB, meaning it can hold up to 5 terabytes of data in total.

2. Is 5TB the standard size for drives over 4TB?

No, 5TB is not the standard size for drives over 4TB. While 5TB drives are common, there are also other sizes available for drives over 4TB, such as 6TB, 8TB, and even 10TB drives.

3. Why do drives over 4TB have different storage capacities?

Drives over 4TB have different storage capacities because they use different technologies and techniques to store data. Some drives may have a higher data density, meaning they can store more data in a smaller physical space, while others may have larger physical dimensions to accommodate more data.

4. Can a 5TB drive store more data than a 4TB drive?

Yes, a 5TB drive can store more data than a 4TB drive. Since 5TB is a larger storage capacity than 4TB, a 5TB drive can hold more data. This is why drives with larger storage capacities are often preferred for storing large amounts of data or files.

5. Is it necessary to have a 5TB drive for storing large files?

It depends on the size of the files you need to store. If your files are smaller in size, a 5TB drive may not be necessary. However, if you have a lot of large files, such as high-resolution videos or large databases, a 5TB drive may be a good choice to ensure you have enough storage space.

Similar threads

Replies
14
Views
2K
  • Computing and Technology
Replies
14
Views
2K
  • Computing and Technology
Replies
11
Views
3K
Replies
2
Views
5K
  • Programming and Computer Science
Replies
4
Views
15K
  • Special and General Relativity
Replies
13
Views
2K
Back
Top