What does disk defragmenting do?

[Kutt]

Turns out that my HDD and SSD are badly fragmented as they have never had a defrag run on them in more than two years. So someone recommended that I run a disk defrag.

What exactly does this do? Will it make my computer run faster?

 

[Andre]

This is what it does. It will speed up all processes that require file reading/writing and that's a lot.

However when it actually runs, it tends to slow down the computer. So it might be helpful to have it run on command, when you're not busy yourself.

Some alternatives.

 

[Bill Simpson]

See if you have any large specific file intensive task that you actually regularly do and which doesn't require finger input while running. Save your original data. Time the task with a stopwatch. Defrag your drive. Then time the task on the original data again. Then please, please, post the results here.

There is folklore that one decibel is about the minimum relative change that a person can detect, unless they are doing the two tasks side by side so they are subtracting rather than actually estimating the time. Google

decibel just noticeable difference

It is my guess that almost nobody will ever see at least a decibel decrease in real actual honest non-contrived work time from defragging their drive. I'd love to see good data on that.

 

[AlephZero]

See if you have any large specific file intensive task that you actually regularly do and which doesn't require finger input while running. Save your original data. Time the task with a stopwatch. Defrag your drive. Then time the task on the original data again. Then please, please, post the results here.

I agree there is a lot of BS talked about defragmentation, but that is a very poorly designed experiment because it doesn't measure how fragmented the drive was beforehand, and unless the data files are significantly larger than the RAM size, most of the data access will be cached in RAM so the disk speed is irrelevant.

On the other hand if you DO have a data-intensive application that continuously accesses data randomly from a large database (hundreds of GBytes) in real time, and where delays of a few milliseconds are obvious without using a stopwatch, quite likely you will see the effect. One such application is computer music generation using a sound sample library. (Of course you don't so much "see" the effect as hear it, as glitches in the audio being generated).

The standard recommendation used to be to defrag the drive before installing the sample lilbrary, to ensure there was a contiguous free area of disk big enough for all the data. But these days, a better solution is to use a solid state disk, where the data transfer rate is only limited by the capacity of the data cables, not by the mechanical design of the disk drive.

 

[Borek]

Does it make a sense to defrag SSD at all? Internal data structure has nothing to do with what we see from the outside.

 

[Bill Simpson]

I agree there is a lot of BS talked about defragmentation, but that is a very poorly designed experiment because it doesn't measure how fragmented the drive was beforehand...

And how would anyone tell a novice, or most other folks for that matter, to carry out an excellently designed experiment in this?

My only goal was: X asks if defrag will help him. I tell him to time it before and after and get some actual concrete numbers that show whether it did or not. (And if you have a "hundred gigabyte audio file" then you have much bigger problem that I could ever hope to address)

 

[Kutt]

I have a 128GB SSD with three games installed on it.

Battlefield 3 with all of it's expansions (over 50GB of space)
Diablo 3
Fallout New Vegas

This drive was 35% fragmented before I ran the defrag. Will this make the games I have installed on it run faster?

 

[Greg Bernhardt]

This drive was 35% fragmented before I ran the defrag. Will this make the games I have installed on it run faster?

That is very high fragmentation. Theoretically, but you may not notice the difference. Maybe it speeds up loading by 4%. That is nice, but you may not really notice it.

 

[Borek]

I will ask again: does the defragmentation of SSD make any sense? I know the logic behind HDD, but SSD?

 

[trollcast]

As far as I have read defraging an ssd will have a minimal effect on speed and because it will cause loads of write accesses to the drive it will shorten the life of the drive.

TRIM is the ssd equivalent of defragging: http://en.wikipedia.org/wiki/TRIM

 

[rcgldr]

I do partition backups via a file / folder copy from a partition to a folder on another hard drive. In this case, defragmenting saves a lot of time. Since the program I created to do the file / folder copy does this in directory tree order, (all files within a directory as a group), then if I do a backup / verify / format / restore / verify, then the restore and any later backups will go much faster as it's almost all sequential I/O operations. This works for Windows XP (using another instance of OS to do the backup), but not for Vista and later (my program doesn't handle junction points yet, but there may be additional issues I'm not aware of). (Note, I also save and retore partition volume id's of partitions since a quick format changes them, which can trigger re-activation or re-install issues).

In the case of a Windows 7 or 8 image backup, defragmentation wouldn't save that much time. I'm not sure if there are any file / folder oriented backup applications available for Windows 7 or 8.

 

[Kutt]

It won't let me defrag my hard drive partition for some reason.

 

[Bill Simpson]

It won't let me defrag my hard drive partition for some reason.

Does it give you any information about why not?
Or does the information it gives just not make any sense to you?
Is your drive more than 90% full? Sometimes that is a reason for Defrag quitting.
Are you using Windows and the built-in defrag?
Have you used or considered the free
http://www.auslogics.com/en/software/disk-defrag/
instead? It warns about nearly full drives, but you can tell it to go ahead and do a less quality job anyway.

 

[Ben Niehoff]

If your drive is very badly fragmented, it might not be possible to defrag it in-place. Defragging moves file segments around, and it needs contiguous blocks of empty space in order to do this. Defragging a highly-fragmented drive is somewhat like playing Towers of Hanoi.

You didn't tell us the error you're actually getting, but if the cause is over-fragmentation, then the only real solution is to copy the data elsewhere, wipe the drive, and copy it back so it will be all neat and tidy. However, as others have pointed out, whether this is actually worth doing is questionable.

Defragging your games drive will not make your games run faster. It might make loading times slightly faster, but that's all.

 

[wuliheron]

I will ask again: does the defragmentation of SSD make any sense? I know the logic behind HDD, but SSD?

SSDs have a limited number of cycles and formating one won't make a significant difference so it is almost always considered counter productive. People who need more speed can buy SSDs that plug into a pci-e slot for faster data transfer rates.

 

[Kutt]

Does it give you any information about why not?
Or does the information it gives just not make any sense to you?
Is your drive more than 90% full? Sometimes that is a reason for Defrag quitting.
Are you using Windows and the built-in defrag?
Have you used or considered the free
http://www.auslogics.com/en/software/disk-defrag/
instead? It warns about nearly full drives, but you can tell it to go ahead and do a less quality job anyway.

No, I have a 2TB HDD and a little more than 100GB of data stored on it. Mostly photographs, music, and miscellaneous software.

My 128GB SSD has battlefield 3 (with all of it's expansions) Diablo 3, and Fallout New Vegas.

It is more than 60% full.

 

[Bill Simpson]

It won't let me defrag my hard drive partition for some reason.

then

No, I have a 2TB HDD and a little more than 100GB of data stored on it. Mostly photographs, music, and miscellaneous software.

So start your defrag program of choice, select the 2TB HDD, click Start and then tell us
1. What defrag program you are using, who made it, what brand is it.
2. Did it start defragging or not.
3. If it did not then exactly what did it tell you.

It is sometimes difficult for a person with a problem to think in a way that they then realize the people on the net hundreds or tens of thousands of miles away cannot see what is on your screen or what the minutes or even months of work have done or not done or what is in your head.

It is very difficult to tell from all the things that you are not telling anyone about this that you tried to defrag the SSD. Perhaps the defrag program is smart enough to simply refuse to defrag SSD drives.

This drive was 35% fragmented before I ran the defrag. Will this make the games I have installed on it run faster?

First, that sounds like you did successfully defrag the drive. True?
Second, since I think some or maybe most gamers are fascinated by whether they are getting 54.13 frames/second or 55.26 frames/second and since you have the game in front of you, tell us, what was the old f/s and what is the new f/s. That is going to be much easier for you to tell us than it is for anyone to psychically remote view your computer and tell you those numbers.

 

[AlephZero]

And how would anyone tell a novice, or most other folks for that matter, to carry out an excellently designed experiment in this?

Hmm... I once was involved in a project to answer a similar question (not specifically about defragging, but how to get the maximum disk transfer rates for an application).

We had both the computer manufacturers and the software developers involved. Since they were trying to sell us the system on the basis of its performance, they were quite interested in the results we were getting (or not getting).

We spent about four weeks getting to the stage where the numbers we were measuring didn't look like random noise. But after that "success", we didn't get anywhere much towards finding the "best" setup, though we did find a few ways NOT to configure the system.

 

[rcgldr]

Hmm... I once was involved in a project to answer a similar question (not specifically about defragging, but how to get the maximum disk transfer rates for an application).

If you have a process where the order that files are read or written in is a known order, then placing those files on a disk in that order will result in the streaming transfer rate from the disk drive, except for the overhead of accessing the cluster information for the files. One example of this is the file and folder copy operation that I use to backup partitions from one hard drive to folders on another hard drive.

At one company I worked for, we wanted to be able to boot from tape, and to simplify this operation, we added a monitor to check the file order during a boot process, and the duplicated that order on the tape. Some files got read more than once, so we have duplicates of those files on tape, so that the boot process was a streaming read from tape without any random access. The additional cleverness was allowing the command interpreter to read it's batch / script lines from the tape, so what ended up on the tape was a booter, command interpreter, then a stream that alternated between batch commands, programs, and data files.

 

[HowlerMonkey]

I used to defrag religiously back in the early 1990s but haven't done it but twice in the last decade.

I notice no performance degradation because of the nature of what I do.

Other uses will be greatly improved by defragmenting such as on my digital audio workstation or anything that requires a good and uninterrupted high sustained data transfer rate.

Sadly, some files "grow" which causes a huge amount of reorganization if they had been previously moved into a space between immovable system files...and defrag decides they will fit better elsewhere.

Proper defragmentation starts right after you finalize your operating system installation and continues throughout the life of the system on the disk.

There are other things you have to consider such as whether you make your swap file static or not so your defrag strategy will change depending on what you are doing with your computer.

 

[clancy688]

Does it make a sense to defrag SSD at all? Internal data structure has nothing to do with what we see from the outside.

No. The SSD controller literally doesn't care on which NAND the wanted data sits. All locations are accesible equally, therefore defragmentating a SSD doesn't make any sense.
To the contrary - since the SSD's NANDs have only a limited number of read/write cycles a defragmentation is very high stress for your disc and will decrease its life time. It's highly recommended to turn automatic defrag and indexing OFF (Windows 7 should do this automatically).
Actually, since there are only a limited number of read/write cycles for every NAND chip of the SSD, its controller is actively scattering data all over the "disc" to distribute the burden to all NANDs evenly. Which means that it's fragmentating data and that this is an intentional process.As for normal HDDs, defragmentation is a good, meaningful thing. If logical coherent data is scattered all over the disc, the disc's reading head needs more time to access the information since it physically has to travel between those two locations. This therefore increases mechanical stress and reading times. So you want to make sure that coherent data isn't scattered. That's what defragmentation is for. But it doesn't make any sense for SSDs where there aren't moving parts anymore.

 

[Borek]

Actually my question was kinda a rhetoric one.

 

[russ_watters]

I will ask again: does the defragmentation of SSD make any sense? I know the logic behind HDD, but SSD?

Not sure if anyone actually answered this, so:

An HDD has read-times dictated by physical motion of a disk and read-write head so if files are fragmented, it takes a lot of physical motion of these parts to read/write to the file, which can take a lot of time. An SDD does not have this limitation: you can access different parts of the device with no time delay in between accesses, therefore there is little or no improvement to be had by defragmenting.

SDD's are random access, and that's one of the main defining features of RAM:

Random-access memory (RAM) is a form of computer data storage. A random-access device allows stored data to be accessed quickly in any random order. In contrast, other data storage media such as hard disks, CDs, DVDs and magnetic tape, as well as early primary memory types such as drum memory, read and write data only in a predetermined order, consecutively, because of mechanical design limitations. Therefore the time to access a given data location varies significantly depending on its physical location.

http://en.wikipedia.org/wiki/Random-access_memory

 

[harborsparrow]

I just posted a lot of details on how to "tune up" a Windows system to help it appear to perform better. See the "tune up" thread: https://www.physicsforums.com/showthread.php?t=678644

 

[harborsparrow]

What operating system are you running? Which partition is your boot drive?

 

[rcgldr]

An SDD does not have this limitation: you can access different parts of the device with no time delay in between accesses.

Even with an SSD, random access does have an effect on performance, specs from an example SDD: 415MB/s Sequential Read, up to 175MB/s Sequential Write and 35K IOPS for Random 4k Read . That 35K IOPS for a random 4KB read translates into 140MB/s transfer rate.

 

[HowlerMonkey]

If you or your defragmentation program are not able to move system files or your virtual memory settings have a dynamic swap/page file, you will always have issues with some files that will always be fragmented.

I said earlier I haven't defragged in a 10 years but all of my systems are/were "ghosted" after a "perfect install" and defrag strategy designed to ensure system files are not fragmented or permanently fixed in places that will cause defragmentation of additional files added to the drives later.

Other guys who did not take this care during install will defrag endlessly and it takes a billion years as the program tries to work with settings that will never allow it to really do a great defrag job.

If you know which files "grow" and/or are replaced by the OS, and properly organize your physical drive, then you defrag only needs to work with the files you are adding to your drive after your perfect install.

Using default settings of your OS and defragmentation program 450 times a year is a huge waste of time and hard drive life.

This is why my hard drives are still buzzing away 16 years later on some systems.

 

[Bill Simpson]

Even with an SSD, random access does have an effect on performance, specs from an example SDD: 415MB/s Sequential Read, up to 175MB/s Sequential Write and 35K IOPS for Random 4k Read . That 35K IOPS for a random 4KB read translates into 140MB/s transfer rate.

I believe you may be confusing two completely different problems here.

I believe the first case makes a single call to the OS asking for a read.

I believe the second case makes thousands of single calls to the OS asking for a read.

As I said several posts ago, show us the numbers. Get your stopwatch. Take an SSD, defrag it, put a gigabyte text file on it, time a gigabyte copy of the file to /dev/null. Then time it with 250,000 calls each copying a 4 kbyte segment to /dev/null. That will measure the overhead of the 249,999 OS calls because your gigabyte file is perfectly defragged. Post the numbers. It isn't hard. But everyone desperately resists putting up the real numbers. (I'd do it but I don't have an SSD or I'd show you the numbers). Then you can try to create a fragmented drive that will satisfy anyone here and repeat the process. That should demonstrate your claim that a fragmented SSD is three times slower than a defragmented SSD. Problem settled.

Show the numbers.

 

[mfb]

See if you have any large specific file intensive task that you actually regularly do and which doesn't require finger input while running. Save your original data. Time the task with a stopwatch. Defrag your drive. Then time the task on the original data again. Then please, please, post the results here.

There is folklore that one decibel is about the minimum relative change that a person can detect, unless they are doing the two tasks side by side so they are subtracting rather than actually estimating the time. Google

decibel just noticeable difference

It is my guess that almost nobody will ever see at least a decibel decrease in real actual honest non-contrived work time from defragging their drive. I'd love to see good data on that.

Even if the difference is not noticeable, it can be useful. You won't see the difference between a 60-minute-task and a 61-minute-task without a clock - but the first gives you 1 minute more for other tasks, with (sometimes) a noticeable result.
On a larger scale, Google (as an example) has large groups of employees to reduce searching times by fractions of a percent, or increase ad click rates by some per mill - too small to notice it without a statistical analysis, but worth millions or even billions.

 

[Bill Simpson]

Even if the difference is not noticeable, it can be useful. You won't see the difference between a 60-minute-task and a 61-minute-task without a clock - but the first gives you 1 minute more for other tasks, with (sometimes) a noticeable result.

And if you take a teacup of water out of the ocean it will help with sea level rise.

And if you save a penny it will help with your $40,000 credit card debt.

And if you overclock your system from 1.8 Ghz to 1.8 Ghz+1hz it will help, even if you don't have the equipment able to measure that.

There is something called "opportunity cost", the price you pay by not doing one thing because you chose to do something else instead. Will spending five minutes defragging your drive provide you enough return to more than make up for that five minutes and be worth more than the other things you could have done instead. My rule of thumb is that if the return isn't significantly well above the noise then I'm just wasting my time. OR you could focus your time and energy on things that will make at least a decibel difference and accomplish far far more.

I remember a friend of mine, now gone, who used to pick up a penny laying in the street. When I said I didn't think that was worth doing he explained "yes, but 100 of them make a dollar" and I replied "yes, but 10,000 of those won't make one of your chiropracter's visits" and he replied "yes, there is that."

I am still betting that defrag will result in far far far less than a decibel change on any SSD and will result in less than a decibel change on a rotating drive under any but the most extreme contrived circumstance.

http://social.technet.microsoft.com...e/thread/c4cca52e-81c3-4139-b600-502b0fbd32fc
says "The automatic scheduling of defragmentation will exclude partitions on devices that declare themselves as SSDs."

http://social.technet.microsoft.com...l/thread/f97425f8-3857-4aa4-9cf5-437d5e212c9c
says "I saw the defrag schedule was enabled on my system too and I have an SSD installed. If you look closer at the defrag utility though and run it, at least in my case, it only takes a few seconds and actually runs the TRIM command on the drive, not the spindle disk type of defrag routine."

 

[rcgldr]

Even with an SSD, random access does have an effect on performance, specs from an example SDD: 415MB/s Sequential Read, up to 175MB/s Sequential Write and 35K IOPS for Random 4k Read . That 35K IOPS for a random 4KB read translates into 140MB/s transfer rate.

I believe you may be confusing two completely different problems here.

This could be related to the high speed ram cache, common with many SSD devices, allowing them to "read ahead" in anticipation of sequential reads, and/or it could be related to the amount of data transferred per command, but the stated specs for don't specify the transfer size per IOP (command) used to produce the sequential read transfer rate.

runs the TRIM command on the (SSD) drive ...

For those that don't know about TRIM, wiki article:

http://en.wikipedia.org/wiki/TRIM

 

[Square1]

See if you have any large specific file intensive task that you actually regularly do and which doesn't require finger input while running. Save your original data. Time the task with a stopwatch. Defrag your drive. Then time the task on the original data again. Then please, please, post the results here.

Do you have particular applications in mind? I haven't defragged in a while and I was hoping to do it this weekend. Keep it simple though I am not super tech literate.

 

[jim mcnamara]

Why has no one mentioned disk latency?

This is usually measured in seek time and rotational latency. For physically spinning disks.

Seek time is the time required to move the disk head to a particular position. Rotational latency is the time it takes for the disk platter to spin to the required position to allow the head to access a disk sector.

SSD's are NOT physical disks. They do not have latency related to the problem of getting the read/write head in the correct physical location on a spinning disk to access a file. And the latency SSD's do have is two orders of magnitude better than physical disk latencies. They are on the order of 1/50000 s. Expensive physical disks have latencies in the 2+ milliseconds range.

Fragmentation is something that affects how a file is physically laid out on a rotating disk. The sector(s) it lives in. If disks are fragmented, then the disk hardware has to "mess around" getting multiple distant sectors under the read/write head. Many seeks can be required to access a file sequentially. Not so an SSD.

Modern disks are much less debilitated by fragmentation. Rotational latency on a SATA III disk at 15000rpm is now miniscule compared to comparable high end drives from the 1990's. Multihead disks reduce seek times.

Plus. Disk controllers cache disk data in controller memory and modern OSes cache file inodes (metadata) plus disk data too in RAM. When a file is opened, with the first read request a large part of the file is read in. My Compellent SAN has a MINIMUM read of 2MB. Solaris is tuned to keep ~100% of inodes since boot in RAM. This is for 100K files.

This explains the comment about "You generally do not need to defrag modern disks".

As aleph zero explained, getting meaningful I/O data and relating it to performance can be trying. And not always fruitful. This is due in part to the obfuscation of real I/O throughput effects by things like the caching I mentioned.

 

[rcgldr]

For both hard drives and SSD's, fragmentation forces the operating system to split up a large read request into multiple commands to handle each fragment. If the fragments are large enough so that command overhead is relatively small, then fragmentation only an issue for hard drives because of the latency as mentioned.

In my case, I perform backups of hard drive partitions using a program that does a file / folder copy of a partition to a folder on another hard drive. I follow this up with a file / folder verify. For restores, I do a quick format (and restore the volume serial number), then copy from folder back to partition and do another verify. The performance of this backup is affected by fragmentation and file ordering, since the program copies files in a fixed order, the directory tree order (all files in a directory copied in one group before handling any sub-directories in a directory). This process is significantly sped up after the first backup / format / restore, since future backups will be nearly all streaming I/O.

Some (or most?) SSD's use internal remapping of data, so what may seem like a defragemented file is still "fragmented" in the SSD's internal mapping. I don't know if a a backup / format / TRIM / restore cycle would "defragment" the internal memory of an SSD or if that would be of any benefit, since a SSD may have an algorithm to equalize usage of it's internal memory via it's internal remapping of 4k clusters.

 

[HowlerMonkey]

Back when hard drives were slow, we made our own custom interleaving to suit our application in the interest of lower latency.

Haven't done it in at least 15 years.

 

[cjl]

Modern disks are much less debilitated by fragmentation. Rotational latency on a SATA III disk at 15000rpm is now miniscule compared to comparable high end drives from the 1990's. Multihead disks reduce seek times.

Interestingly enough, this simply isn't true.

First of all, modern SATA disk drives run at 5400-7200 RPM. A few (the WD Velociraptor series) run at 10,000 RPM. None run at 15000 RPM (15k drives do exist, but they run on a SAS interface rather than SATA, and are pretty much exclusively for enterprise use). Multihead makes no difference to seek time (just look at the seek times for drives of differing capacity that are the same generation - most modern 500GB drives are 1-2 head, while 2TB drives have 4-8 heads). Most modern hard drives have a total access time on the order of 12-18 milliseconds, with the very best consumer drives (the WD Velociraptors mentioned above) getting as low as 7ms or so. The Quantum Fireball EX drive from 1998 (available in capacities up to 12 GB!) had an access time of around 16 milliseconds. Yes, the better modern drives are closer to 12ms, but that's hardly miniscule by comparison.

The bigger factor in why modern drives don't need to be defragged as much is a combination of 2 factors. First, hard drives have a much larger cache and systems have much more RAM. As a result, much more user data is cached, and this somewhat isolates the user from the direct performance impact of the disk itself. Second, the operating system is much more intelligent about how it places data - Windows 7 is much better about preventing fragmentation in the first place, whereas Windows 95 or 98 was much more simplistic in its methods of drive access.

 

[harborsparrow]

I have found that there is one case in which it is very useful to defragment Windows machines, and that is immediately after they are installed. Once this initial defrag is done, then blow up the swap file (in the Control Panel options--message me if you want to know how) so that it is already at its maximum size (set the minimum and maximum size of the swap file to the maximum recommended size.). This avoids having the swap file shrink and grow over the years. Providing one large, contiguous allocation for the (invisible to most users) swap file used by the operating system will save you the pain of the system eventually slowing down if the swap file gets broken into multiple pieces that lie far from each other on the disk.

This is especially true if the operating system (and all the files on it) take up a large percentage of the drive.

I've studied computer architecture and memory systems at great length over the years, and I've had a ton of practice with this, and I think this holds even for SDD's--getting that one large allocation for the swap file will help Windows stay tuned up.

Other than this, I doubt it's all that important to defrag the more modern systems even if they have moveable drive parts, because the disks are typically huge as compared with the amount of files on them, which means the drive will just keep allocating new, unfragmented space for a darn long time. Besides, most Windows systems these days will defrag automatically (if you don't stop them) in the background when you are not using the machine. Usually, I tell Windows not to defrag on a schedule or at will, and I do it myself once in a while (usually while watching a ballgame or overnight).