Can you restore data from a deleted file that was previously emptied?

[jack action]

TL;DR Summary

Does erasing the content of a file before deleting it offer any advantages to making the content unrestorable?

Say you have a large text file on your hard drive that you edit such that its content is fully erased and you save it that way. Then you delete the file. Is the content still on the hard drive?

Usually, deleting a file only deletes the address where the file is on the hard drive and the content of the file can be restored as long as nothing is written over it. But in the case of erasing the content before deleting the file, can the content still be retrieved? If so, would it at least make it a little bit harder to find or would the computer overwrite the erased content faster?

 

[PeterDonis]

Say you have a large text file on your hard drive that you edit such that its content is fully erased and you save it that way. Then you delete the file. Is the content still on the hard drive?

You need to specify more exactly what "fully erased and you save it that way" means. The specific answer might well be application dependent.

If you mean you are editing a text file, and you delete the entire file in your editor so it's empty and then save the file, that would not erase the file's data on disk, it would just release the blocks where the file's data was previously stored into the free pool, which means the OS would be free to overwrite the data if it needs to store new data. But it could be a while before that happens (it would depend on how full your disk is and how much file saving you do).

But not all applications work the same way. Some applications write empty data to disk in place of the old data (meaning, they write to the same blocks on disk where the old data was stored) when you "erase" something. In those cases, the old data would be gone.

Usually, deleting a file only deletes the address where the file is on the hard drive

More precisely, it deletes the tracking entry that points to where the file's data is. Even that is not necessarily straightforward. On Linux, for example, deleting a file just deletes a directory entry; it doesn't necessarily touch the file's inode, which is where the pointer to the actual data is located. The inode gets deleted only if all directory entries that point to it have been deleted.

And even that isn't always the case, since many systems have a "trash can" or something like it, so "deleting" a file doesn't actually mean deleting its directory entry, just moving that entry from the previous directory to the "trash can" directory. To the OS, the file is still pointed to by a directory entry and it still exists as a file. You would have to empty the trash can to actually delete the file as described above.

the content of the file can be restored as long as nothing is written over it.

If you can find the data, yes. The easiest case is if the blocks on the disk that stored the file's old directory entry have not yet been overwritten; then a file restore program can just look at the old directory entry and find the file's data.

It gets harder if the old directory entry has been overwritten by newer data, but it's still not impossible. File restore programs have various tricks to try.

 

[FactChecker]

Usually, deleting a file only deletes the address where the file is on the hard drive and the content of the file can be restored as long as nothing is written over it. But in the case of erasing the content before deleting the file, can the content still be retrieved?

It can still be retrieved unless serious efforts were made to erase the data. Certainly, the FBI has tools to recover files that were simply modified once and deleted. I don't know who else would have those recovery tools.

If so, would it at least make it a little bit harder to find or would the computer overwrite the erased content faster?

Yes, it makes it harder. A casual effort would not recover the data. But If you want to do a better job of erasing data, I recommend using a tool like Eraser by Heidi. It will overwrite a file several times with random bits. There are several options for overwriting the files:

A final option is the total physical destruction of the drive.

 

[Filip Larsen]

Not sure if anyone has mentioned this yet, but for SSD's overwriting the content does not usually delete the original data just writes as similar number of blocks in a different place.

 

[FactChecker]

Not sure if anyone has mentioned this yet, but for SSD's overwriting the content does not usually delete the original data just writes as similar number of blocks in a different place.

Exactly. I'm sure the same is true of any hard drive. An eraser tool would have to go to the physical blocks of the current file and directly write random bits there.

 

[Filip Larsen]

I wonder if some SSD vendors have tools that allow security wipes of either the whole disc or perhaps even selected files or similar? I assume it requires a low-level tool to map and overwrite the physical blocks in use.

 

[StevieTNZ]

@jack action - are you hoping for data recovery of a file, in this instance?

 

[FactChecker]

I wonder if some SSD vendors have tools that allow security wipes of either the whole disc or perhaps even selected files or similar? I assume it requires a low-level tool to map and overwrite the physical blocks in use.

Windows has a full reformat of a drive. It's my understanding that their quick format just cleans out the directory. It takes a long time to do a full reformat a large drive.
The Eraser tool that I referenced above allows you to overwrite the blocks of an individual file many times. And you can make Eraser an option for the files in Windows Explorer.

 

[FactChecker]

Don't forget about backups of the file. In general, I recommend that any sensitive information be kept in an encrypted drive with a strong password. That way, your backups are as well protected as the original. I use VeraCrypt by IDRIX for that.

 

[PeterDonis]

I'm sure the same is true of any hard drive.

Not quite. SSDs work differently from previous hard drives. On previous hard drives, although many applications, when "overwriting" a file, actually just write new data to different blocks and then change the file's inode (or whatever the Windows equivalent is) to point to the new blocks instead of the old, it is also perfectly possible, either for applications or the OS itself, to explicitly write new data to the same blocks. (Many database engines, for example, explicitly control writes at the block level, bypassing the OS's filesystem driver.)

On SSDs, however, there is no way for an application or an OS to explicitly write new data to the same blocks. From the application's or the OS's point of view, every write writes a new block. The drive's internal firmware can overwrite blocks in order to garbage collect old blocks that no longer contain active data, but this is not done unless absolutely necessary, in order to preserve the NAND flash memory on the drive as long as possible. There might be some special tool that can force the drive to do this to purposely erase data, but it's not something that is ordinarily done the way applications and OSs could with previous hard drives.

 

[Filip Larsen]

A workable approach for wiping a SSD drive if this requirement is known before filling it with data is to use it only with full disk encryption. This means the drive (or partition in question) is effectively wiped if one can wipe just the master key. Alternatively one can make a logical encrypted disk in a regular file system using a tool like truecrypt (or whatever pose as its replacement these days - truecrypt was marked as Not Safe Anymore by its maintainer some years ago).

 

[jack action]

The thought came up to me as I often write to-do lists in a txt file and I delete the items once they're done. In the end, I sometimes end up deleting the last item leaving an empty file that I delete afterward. I was wondering if deleting the last items before deleting the file made a difference with the data stored on the disk.

I was thinking that the program used to modify the content may affect the process. But let's keep things simple. Open a file with python and put an empty string in it then close the file. Do we really know what happened to the old data? What if I do the same thing with Java instead? What if it is not done with the same OS? Are different programs really using different approaches to how the files are saved? (Not talking about programs made especially for data security.)

Or is it really a new file that is written each time I save it, meaning I have all of my previous versions saved up somewhere until they are written over?

 

[PeterDonis]

Open a file with python and put an empty string in it then close the file. Do we really know what happened to the old data? What if I do the same thing with Java instead? What if it is not done with the same OS? Are different programs really using different approaches to how the files are saved?

Most programs or programming languages are probably using the standard library functions provided either by the OS or the C library for the OS. (I know Python does.) You would need to look up the documentation or source code for those functions to know for sure exactly what they do.

 

[FactChecker]

The thought came up to me as I often write to-do lists in a txt file and I delete the items once they're done. In the end, I sometimes end up deleting the last item leaving an empty file that I delete afterward. I was wondering if deleting the last items before deleting the file made a difference with the data stored on the disk.

It makes some difference. A casual attempt will not get the old version, but a serious effort can get it.

I was thinking that the program used to modify the content may affect the process. But let's keep things simple. Open a file with python and put an empty string in it then close the file. Do we really know what happened to the old data? What if I do the same thing with Java instead?

Two different languages running in the same operating system, using high-level system calls, probably do the same thing. That being said, some languages run in a "sandbox" and might interact with the hardware differently.

What if it is not done with the same OS?

Using a different OS can make a difference.

Are different programs really using different approaches to how the files are saved? (Not talking about programs made especially for data security.)

Or is it really a new file that is written each time I save it, meaning I have all of my previous versions saved up somewhere until they are written over?

It is more likely that the decision of what to do when a file is changed is done a block at a time. I doubt that a small change in a huge file would cause a rewrite of the entire file.
I think that it would take a lot of investigating to determine the answers to all these questions and the answers might be different for different files, languages, OS, hardware devices, and device handlers.

 

[PeterDonis]

I doubt that a small change in a huge file would cause a rewrite of the entire file.

I'm not so sure. Applications in general don't view files as sequences of blocks, but as sequences of bytes. (There are some exceptions, such as the database engines I mentioned before; but a text editor, for example, views files the way I have described.) If the disk has 512 byte blocks, and a change is made that inserts a byte after byte 500 of the file (so bytes 501 and on are now different from what they were before), saving the file in a text editor will result in every block of the file being rewritten to disk, since every block's bytes have changed.

 

[FactChecker]

saving the file in a text editor will result in every block of the file being rewritten to disk, since every block's bytes have changed.

I'll buy that. I suppose this is especially true if the editor encrypts the file, as I have been doing for sensitive information.

 

[Filip Larsen]

saving the file in a text editor will result in every block of the file being rewritten to disk

Well, a program doing, say incremental save of a textual file, don't have to update the whole file but could resort to use seek and only write from the first actual change. That is often how logger libraries work to avoid having an O(N²) performance as the file grows.

There are even file systems that allow you to seek and write content with holes, i.e. the file system has the option to skip (some) blocks earlier in the file that has never been written yet.

But back to the question. If I had to use python to write and later read a file (i.e. a file not needing to be read by say a text editor) which I want to be able to securely wipe afterwards even if the file resides on a SSD or flash drive, then for I would consider to use encryption. Even the seek and write pattern of a logger can be used with block ciphers at least if a suitable block cipher mode is selected.

 

[Vanadium 50]

The application should neither know nor care about disk formats. You do not want tp update your app when plugging in a slightly different USB sticlk.

Security needs to be balanced - what are you protecting and against whom. Protecting the
honeydew list" one's spouse gave you from attacls by major world governments might be overkill. ("Comrade Boris! Target is painting ceiling beige!")

People want teir data to be erased permanently and irrevocably - unless they erased it by mistake. Then they want ir immediately and easily recoverable. This is not easy to do.

My Linux filesystem does writes by making copies, and when the copy is known to be good, the original is marked for deletion. It makes copies at the block level. However, for a honeydew list, I bet one file is one block.

 

[FactChecker]

Security needs to be balanced - what are you protecting and against whom. Protecting the
honeydew list" one's spouse gave you from attacls by major world governments might be overkill. ("Comrade Boris! Target is painting ceiling beige!")

Ha! Good point.
That being said, I would encrypt financial data such as tax returns. Encryption has the advantage that you can do backups of everything without worrying about the financial data on it. Relying on deleting unencrypted files would require deleting those files on any backup media.

 

[PeterDonis]

a program doing, say incremental save of a textual file, don't have to update the whole file but could resort to use seek and only write from the first actual change.

Yes, that's why I specified, in my example, that the change was in the first block, so every block's data would change. The application won't actually know that; all it knows is that it is saving the file's data from byte 510 on (if it uses the incremental save method that you describe, which many editors do). But the filesystem driver will end up actually writing to disk in blocks, and every block's data will be different from what is currently on disk.

 

[jack action]

But the filesystem driver will end up actually writing to disk in blocks, and every block's data will be different from what is currently on disk.

But would it write it on different blocks or overwrite the original blocks (even if there are enough blocks available somewhere else)?

 

[PeterDonis]

would it write it on different blocks or overwrite the original blocks

I would expect most filesystems and OSs to write it on different blocks. That minimizes the chance for corruption of data since the operation of actually changing the file data then reduces to just updating pointers in the inode (or whatever the Windows equivalent is); you're not actually overwriting any data, you're just shifting pointers from the old data to the new. That means there is never a time when a complete, consistent version of the file's data is not on disk--it's the old version until all of the new blocks are written and the pointers updated, then it's the new version.

 

[Vanadium 50]

But would it write it on different blocks or overwrite the original blocks

The application should neither know nor care. The OS and FS determine this,.

On my system, what happens is (silightly simplified) is that the new data is written to empty space and checked. If it passes the check, then the pointer to the data is updated from the old data to the new data. This effectively marks the old space as available for new writes.

But there are many other options.

 

[Vanadium 50]

I would expect most filesystems and OSs to write it on different blocks

Modern, probably. No good reason to do otherwise. Not so much the older ones. Then you had downsides - fragmentation was an issue, and available space was a bigger issue. And, as you might expect, corruption was a bigger issue then.

 

[FactChecker]

You certainly can NOT COUNT on the same block being used. In a modern SSD, the writes are spread out so that no part is used excessively. That extends the useful life of the SSD. I'm not sure if there is any way to control that at the level above the SSD device.

 

[Vanadium 50]

I'm not sure if there is any way to control that at the level above the SSD device.

There is, but why would one want to?

One allocates the entire SSD to one big file and fills it. To write a block, one needs to erase a block, and one can always pick the block to erase. Now there is only one unused block so that's where the write happens.

But why would anyone do that? "What it loses in speed it gains in unreliability."

 

[Filip Larsen]

One allocates the entire SSD to one big file and fills it.

Not sure that will work on modern SSD. As far as I am aware, they have more physical blocks than logical available and will automatically remap "worn out" (nearly bad) blocks to new "fresh" blocks in order to maintain available disk space over time and without having to perform OS-level disk recovery. I assume such a "retired" block is near impossible to read from the OS-level but I also assume most vendors will have a analysis or recovery tool available that likely are able to read most data in those blocks anyway.

 

[.Scott]

But would it write it on different blocks or overwrite the original blocks (even if there are enough blocks available somewhere else)?

It depends hugely on the operating system, the version of that operating, and the type of disk media.
In some cases with spinning media, updated data will overwrite existing blocks - thus reducing file fragmentation. But you certainly can't depend on this.

I am not on a Windows system right now, so I can't check it out, but there used to be a DOS command "cipher" that would perform an overwrite of unused sectors on any NTFS file system. It would overwrite the unused sectors with 5 different patterns so that the data on spinning media could not be recovered even with the most sophisticated techniques.

cipher /w:c:

 

[sbrothy]

Just FYI, POSIX has this one:

shred

"Overwrite the specified FILE(s) repeatedly, in order to make it harder for even very expensive hardware probing to recover the data."Beware of the CAUTION mentioned though.

 

[pbuk]

Just

Look at the copyright on the link: 2016. In 2023 journalling file systems and SSD drives are ubiquitous. If you want to destroy (rather than erase) data, destroy the drive. In a fire.

 

[256bits]

Windows has a full reformat of a drive. It's my understanding that their quick format just cleans out the directory. It takes a long time to do a full reformat a large drive.
The Eraser tool that I referenced above allows you to overwrite the blocks of an individual file many times. And you can make Eraser an option for the files in Windows Explorer.

Does a full reformat write Ctrl-Z's and re-read ( or what ever the operating system uses for null data ) to the entire disk?
Or does it just read and re-write the sector information, and cluster size it.
Some older systems did not re-write the whole disk with full format, and just took samples of sector data to verify good/bad bits/sectors.

 

[sbrothy]

Look at the copyright on the link: 2016. In 2023 journalling file systems and SSD drives are ubiquitous. If you want to destroy (rather than erase) data, destroy the drive. In a fire.

I'm obviously a dinosaur.

Incidentally, I had an extremely disgusting talk with a self-confessed pedophile when I was a moderator on a filesharing network (DC++) a long time ago (I knew which keywords to look for, I never downloaded anything from him.) I soon had the authorities on the phone trying to track him down but he was a slippery customer. They never found him even though I willingly gave them all the data I had. They didn't care one bit about our piracy.

Point is:

He claimed he had a remote controlled magnesium block rigged to burn lying on top of his harddisk-stack.

Talk about a nasty experience. Obviously I don't know if he was telling the truth but he seemed very smug and perversely proud of himself.

I had to take a very long shower afterwards.

 

[FactChecker]

Does a full reformat write Ctrl-Z's and re-read ( or what ever the operating system uses for null data ) to the entire disk?
Or does it just read and re-write the sector information, and cluster size it.
Some older systems did not re-write the whole disk with full format, and just took samples of sector data to verify good/bad bits/sectors.

I'm unsure of the details, but several people (of unknown expertise) say that the full format makes the data unrecoverable. (see e.g. this ). The full format certainly takes a very long time as though something is being written to the entire disk.

 

[Nugatory]

I'm unsure of the details, but several people (of unknown expertise) say that the full format makes the data unrecoverable. (see e.g. this ). The full format certainly takes a very long time as though something is being written to the entire disk.

There’s a difference here between magnetic media and flash. Flash, once physically overwritten, is truly gone. Magnetic media can be overwritten several times but a well-funded attacker (in practice, that means a national intelligence agency) with a scanning electron microscope and possession of the media will still be able to recover much of the old data.

 

[Vanadium 50]

In a fire.

Drill press!

I will make the same comment I have elsewhere - are you trying to keep your little sister out, or major world governments out?

If the latter, I would reply with two statements:
1. You're not that important.
2, Never, ever, ever write anything to the drive that isn't encrypted.

 

[pbuk]

Flash, once physically overwritten, is truly gone.

This is true, but irrelevent. On an SSD, data at the hardware level is controlled by the device: there is no way to control the process in software so you have no way of knowing whether any particular hardware location has been physically overwritten. After dissassembling the device a specialist can access the hardware, however

1. You're not that important.

 

[Nugatory]

This is true, but irrelevent.

Not completely irrelevant, but yes, you need to make some assumptions about the wear-leveling algorithm. I would be somewhat surprised to find that writing the entire block address space many tens of times would leave any data-containing physical block untouched…. But then again in a previous life I found drive firmware doing many strange and wonderful suboptimal things, some of which were unknown and surprising to the drive manufacturers themselves.

 

[TonyStewart]

(in practice, that means a national intelligence agency) with a scanning electron microscope and possession of the
media will still be able to recover much of the old data.

MYTH #1:
This assumption that overwritten data is recoverable is extremely unlikely. Perhaps some bits can be read but not words, sentences or sectors.

The HDD magnetic overwrite specs used to be -20 dB min, it may be less now, but applying Shannon-Hartley's formula for the bit error rate, BER with the Signal/Noise Ratio, SNR and bit shift which I have personally done on HDD's with spectrum analyzers for 1f, 1.5f, 2f etc the old data is well below a reasonable probability of recovering any sectors or even strings of text over-written with RLL code.

However there exist algorithms based on MFM back in the old days for random data over-write several times. There used to be single servo surface control and other surface might have more residue from old data if off track written at a different temperature. Yet for a long time now servo has been dedicated on each track and surface with hard-sector servo dibits between tracks to eliminate the off-track errors from old technology.

 

[Nugatory]

This assumption that overwritten data is recoverable is extremely unlikely. Perhaps some bits can be read but not words, sentences or sectors.

You are considering recovery by imaging the surface of the platter with a scanning electron microscope? I’ve seen SEM images in which the magnetic domains at the edge of the track are clearly visible, and they looked very different when a write changed the value instead of putting down the same value.

But of course this was older drive technology, decade before last. I can believe that writes now leak much less information at the edge of the track.

 

[TonyStewart]

Yes we also had an SEM at Burroughs for magnetic recording experiments and I once qualified suppliers for heads with specs for ATI (adjacent track interference), OW (Overwite) with dB suppression, track and bit resolution specs. it's still a myth.

I made an interesting experiment where I would use the PES position error signal and inject an error from the Y axis output of a scope and then using the X axis ramp on Y Ch1 I could scale any offtrack pattern I wanted like +/-10% then write 1f then repeat with 2f and look for 1f residual on a spectrum analyzer with the output synchronized to the scope which was synchronized to the RPM index pulse of the spindle. So I could get a transfer function of overwrite residual vs offtrack ramp wave or adjacent offtrack interference. Then I could adjust the magnitude and offset of the heads using Ch1 DC and gain. This way I could measure SNR in dB as a function of any disturbance.

 

[Vanadium 50]

Burroughs

Good heavens - they haven't been around in what, 35-40 years?

In those days. weren'y drives powered by steam?

 

[TonyStewart]

Good heavens - they haven't been around in what, 35-40 years?

In those days. weren'y drives powered by steam?

they did consume massive amounts of power in our Winnipeg plant , I had a dozen mainframes, a dozen ATE's , 2 clean rooms and hundreds of products while testing all the OEM's like Maxtor Hitachi, Seagate, Toshiba from the inside out, so I invented new test methods as TE Mgr while my staff wrote procedures and setup all peripheral types in production. We also were Air Canada's biggest customer and I had over $10m in test equipment. and that was only my 3rd career in high tech after Aerospace and Telecom.

 

[Vanadium 50]

On an SSD, data at the hardware level is controlled by the device

HDDs - regular spinning ones - also do it, to a lesser degree. If the firmware think a sector is questionable, it will remap it. The old data is still on the platter, just not visible to the user.

If you don't want people looking at your daat on your drive when you throw it out, mechanically break it first. Not good enough? Then encrypt it when you get it and ensure that unencrypted data never touches it. Still not enough? Seek professional help.

 

[TonyStewart]

A low level format is adequate, a secure format is for those who prefer. A hammer to the lid is effective. They can withstand significant levels of extreme magnetic forces without erasure in the sealed unit.

 

[Vanadium 50]

It's not the time of Burroughs any more, and it is not clear to me what a "low level format utility" does today. They exist, so they surely do something, but it's not like the old days - for instance, they can't change capacity (downward, obviously)

They promise a lot of questionable things - make your drive faster! Make it last longer! So far as I can tell, they will zero-erase the drive's sectors via the firmware. Whether this includes formerly reallocated sectors is unclear.

But let's step back a moment. People often replace drives because they are failing - or have failed. By then, it may be too late to erase things. That's why I argued for encryption before getting this far along.

 

[TonyStewart]

“zero-filling” is essentially the modern-day version of the old process of “low-level formatting” and that you should be completely certain that you’re happy to lose the data on your hard drive forever when you do it.

Windows might have a empty box for [ ] quick-format.
Regardless of the soft format, modern drives use 4k hard sectors with 100 B of ECC then 16 B sector addressing and inter-track servo di-bits.

I only stated multiple pass over-writes was designed for older dedicated servo technology and still available today, but my experience with embedded servo drives today, is that it is overkill.

This has nothing to do with alternative methods of dealing with security.

Multiple pass random data over-writes: Not required

(It's paranoia.)

 

[.Scott]

Here's the advice from the CISA (Cybersecurity and Infrastructure Security Agency).
CISA is part of the US Department of Homeland Security.

If a magnetic drive contains US National Security Information, it requires physical destruction of the magnetic platters before disposal.

 

[Vanadium 50]

it requires physical destruction

Drill press!

 

[sbrothy]

TL;DR Summary: Does erasing the content of a file before deleting it offer any advantages to making the content unrestorable?

Say you have a large text file on your hard drive that you edit such that its content is fully erased and you save it that way. Then you delete the file. Is the content still on the hard drive?

Usually, deleting a file only deletes the address where the file is on the hard drive and the content of the file can be restored as long as nothing is written over it. But in the case of erasing the content before deleting the file, can the content still be retrieved? If so, would it at least make it a little bit harder to find or would the computer overwrite the erased content faster?

I mentioned the linux command "shred" in this thread but was told it's out of date. Depend on the drive type was the point I think.

 

[Rive]

I mentioned the linux command "shred" in this thread but was told it's out of date.

I think you have accidentally linked this very thread

BADBLOCKS in destructive mode will do for old style drives.