Is secure deletion possible in Java

[Paul Uszak]

Let a file on disc comprise the bytes "Hello World".

If I open the file in Java as a RandomAccessFile and write the character "X" over it from the start of the file to it's end, will the original disc block be overwritten? Or, will the original block containing "Hello World" be marked as unused, and another block written as "Xello World" and so forth till a final block is written containing "XXXXXXXXXXX"?

Is it possible to do secure deletion entirely in Java? (...I'm aware of secure deletion programs)

 

[phinds]

I'm pretty sure that actual file access is done from O.S. subroutine calls and it is irrelevant what language you use to create the machine language call. The O.S. will open the same file and, yes, overwrite it.

 

[Paul Uszak]

The O.S. will open the same file and, yes, overwrite it.

Do you mean overwrite the same physical magnetic block on the hard disc surface as laid down by the drive's firmware? So that block will be written to 11 times?

 

[phinds]

Do you mean overwrite the same physical magnetic block on the hard disc surface as laid down by the drive's firmware? So that block will be written to 11 times?

No, why would it be done 11 times?

If you write the file once, it will be written once. If you want it to overwrite 11 times, you have to specifically tell the O.S. to write the file 11 times. I'm not aware of any O.S. sub (at least under Windows) that writes a file 11 times with one call.

 

[Filip Larsen]

Is it possible to do secure deletion entirely in Java?

Yes, in the same way as in any programming language with access to file random access. However, this method of secure erase would only apply for HDD (Hard Disc Drive) or other storage where the location of the file block retain their position. For this to be effective you would also have to make sure that the blocks are written many times during the process and not cached in RAM by the OS and then just written once.

For the modern SSD (Solid State Drive) overwriting a file for security measures does not make sense as the blocks of the files are changing on the device when blocks are updated and written by the OS. For files on SSD's you would need other (low level) methods [1] for securely erase a file, methods that to my knowledge is not yet available on the OS file API level less alone from Java. To the extend the SSD in question supports such secure erase, it would most likely need a special vendor or BIOS tool to securely erase a file or whole partition from that disc.

Later: I can see in the description for the SysInternal sdelete tool [2] that they apparently only overwrite files once using a certain bit pattern. If one overwrite is all it takes, my concern stated above about the OS caching blocks between multiple overwrites does of course not apply.

[1] https://en.wikipedia.org/wiki/Solid-state_drive#Data_recovery_and_secure_deletion
[2] https://technet.microsoft.com/en-us/sysinternals/bb897443.aspx

 

[phinds]

For this to be effective you would also have to make sure that the blocks are written many times during the process and not cached in RAM by the OS and then just written once.

Excellent point. I hadn't even thought about that.

 

[Carno Raar]

For the reasons explained above (SSDs, OS leaking info via swap files etc), if you are depending on secure deletion you might additionally wish to look into full disk encryption.

 

[Paul Uszak]

...you might additionally wish to look into full disk encryption.

Can't really rely on encryption unfortunately. My scenario calls for a program that will run on varied machines. It will then securely delete a part of itself when the program has been run. I was wondering if this could be achieved solely in Java?

I can't rely on them having /being able to have disc encryption.

 

[rootone]

Any programming language should be able to delete or overwrite files that it created.
I don't see the problem.

 

[Paul Uszak]

Any programming language should be able to delete or overwrite files that it created.
I don't see the problem.

From my question title, I was referring to secure deletion.

 

[rootone]

So before you detete the file you replace it's content with random gibberish.

 

[Paul Uszak]

So before you detete the file you replace it's content with random gibberish.

and this will overwrite the same physical magnetic block on the hard disc surface as laid down by the drive's firmware? You realize of course that erased data isn't really erased?

 

[rootone]

A block of data on a magnetic disk is what it is, bits of information stored and retrievable by means of atomic magnetic orientation.
As far as I know computer disk magnetism is 'at it is' and does not contain any memory of it's previous content, but I am willing to hear of how this could be possible.

 

[FactChecker]

A block of data on a magnetic disk is what it is, bits of information stored and retrievable by means of atomic magnetic orientation.
As far as I know computer disk magnetism is 'at it is' and does not contain any memory of it's previous content, but I am willing to hear of how this could be possible.

There is no guarantee that the modified file is written back to the same place on the disk. So the original data can still be there. This is certainly true for sequential access files and the question is whether it is also true for direct access files. Even for direct access files, there is some chance that a disk sector will be marked as bad and would not be overwritten. I doubt that there is any way, using high level disk access, to guarantee that a particular sector is overwritten. It's possible that the only sure way would be to copy all other data to another disk and reformat the entire original disk.

 

[rootone]

Sure, but it's definitely possible to write whatever gibberish you want to a specific disk block if you have have reason to do that.

 

[phinds]

I'm not sure if it's still true but in the early days of Windows, the carry-over from DOS was that there were system calls that let you access the file system directly via the File Allocation Table and you COULD be sure that you were overwriting what you wanted to be overwriting. File systems are a bit more complicated nowadays so that capability may no longer exist in windows. I haven't kept up with such things.

 

[phinds]

Sure, but it's definitely possible to write whatever gibberish you want to a specific disk block if you have have reason to do that.

Only if the O.S. allows you direct access to the file allocation system, otherwise you have no way of knowing whether you are writing to the same place on the disk or not. You seem to be conflating high-level disk operations with low-level disk operations in a way that may not hold.

 

[phinds]

As far as I know computer disk magnetism is 'at it is' and does not contain any memory of it's previous content, but I am willing to hear of how this could be possible.

I'm with you on this one, but there have always been people who are paranoid about security who insist that it is possible. That has never made the slightest bit of sense to me but I've never delved into the technical details so I cannot say w/ 100% assurance that it can't be done. There are numerous app, and pretty much have always been (since the early days of PCs) that "guarantee" what they call "secure" overwriting, which means writing to the same exact magnetic bit over and over and people buy them or they would not still be around.

 

[Filip Larsen]

My scenario calls for a program that will run on varied machines. It will then securely delete a part of itself when the program has been run. I was wondering if this could be achieved solely in Java?

This is in general not possible to do securely in the sense that it would be very easy for an adversary to break this by denying your process the permission to overwrite the files in question or by making copies of your program and restore them after your program deletes itself. Note that using a virtual machine it is very easy to save and restore the full HDD state of a machine to one (or many) points in the past simply by the click of a button.

A long story short, you cannot trust or rely on anything deployed to a client machine you have no (physical) control of. If you need some kind of license control of your system the "most secury" route dictates that you move part of your licensed service (i.e. the functions your program perform) to a host that you control. The logical end-point of this argument with current technology is that if you need strong license control you would probably want to use a cloud solution and only use the client for presentation and use of non-licensed functions.

 

[Carno Raar]

My scenario calls for a program that will run on varied machines. It will then securely delete a part of itself when the program has been run. I was wondering if this could be achieved solely in Java?

What's to stop them having the JVM pause and dump your code? There's even an open source JVM which can be made to do anything with your bytecode.

I don't know high profile examples from Java, but for other VM languages, you can look at Dropbox's attempts to obfuscate Python (they used a custom interpreter) and Zend's attempt to build a PHP obfuscator using encrypted bytecode. It doesn't work, because, eventually, you have to run regular bytecode and the intepreter or JVM will give you up when sufficiently prodded.

You're going to need a language that runs on the metal or you going to need to fork the JVM and add obfuscation into it. You can't run regular Java bytecode thats's binary compatible with a standard JVM. You could send the data over the internet to a private server you control. Run your secret algorithm on that, and send the data back.

I'm with you on this one, but there have always been people who are paranoid about security who insist that it is possible. That has never made the slightest bit of sense to me but I've never delved into the technical details so I cannot say w/ 100% assurance that it can't be done. There are numerous app, and pretty much have always been (since the early days of PCs) that "guarantee" what they call "secure" overwriting, which means writing to the same exact magnetic bit over and over and people buy them or they would not still be around.

In the old days they felt you could use electron microscopes and similar tools to read the residual signatures from the disk, and you could, so various governments therefore drew up secure disk wiping schemes like the famous 7-pass that's used by US Government. However, newer drives have a much higher density of data and we currently believe it's not possible to extract data meaningully from a disk that's been zero filled once. It's known various intelligence agencies were unable to recover data from the wiped portions of Edward Snowden's Macbook. If you need another name to google w.r.t. secure erase, there is some work by Gutman. I believe he was the original author of the '7 pass' paper.

Of course with SSD its more complex as they can reorganise data with their internal controller so you have no guarantee that if you write to the same spot on the disk twice, that you will actually get the same physical cell both times. However, a zero fill of the entire drive does work. To be honest I am not sure how SSD do it, but I know you can't rely on file-level zero-fills on an SSD. It's got to include freespace.

The current recommended way to securely erase data when a PC is destroyed is an industrial shredding machine or thermite. It's just faster than any kind of erase. If you're reselling a machine, boot from a Linux USB stick and run "sfill". You can set the paranoia level in options. Personally I run full disk encrpytion on all my stuff, so even if someone does steal it, they ain't going to get any data off it unless they hit me with a pipe until I give up the decryption keys.

 

[phinds]

Do you happen to know how much your performance is degraded by the encryption? Seems it would be significant if you are doing a lot of disk I/O

 

[Carno Raar]

Do you happen to know how much your performance is degraded by the encryption? Seems it would be significant if you are doing a lot of disk I/O

Not personally.

However, here are some benchmarks. His figures give a 5% increase in CPU activity.

Phoronix Ubuntu FDE benchmarks: http://www.phoronix.com/scan.php?page=article&item=ubuntu_1404_encryption&num=1

Since you are IO-bound anyway this probaby doesn't mean anything significant for the average user. 5% more cpu ticks is nothing compared to a disk read. I don't really know to interpret that 5%.

 

[Paul Uszak]

My application would be used consensually, so there shouldn't be an "adversary" as such. It's not licensing either. It's a comms application that deletes one time pad data pertaining to messages sent. It's meant to serve two purposes:-
a. prevent the user unintentionally resending a message twice with the same random keys
b. impede the Man from getting access to the keys for messages sent

So it's to the user's advantage to not set up a hostile environment for the application. Point (b) is harder to deal with it seems... I'm trying to access the viability of measures to protect the one time pad data. I just know that someone's going to suggest using a password. Passwords unfortunately can be "extracted" from users.

My initial idea was along the lines of "self destructing" the data. I might have to fork a lower level secure deletion program perhaps, if Java can't do it itself.

 

[Carno Raar]

What's wrong with PGP (or GnuPG)?

Although if this is a research project, by all means go ahead and try to get one-time pads working. I won't say don't do research: you may be the one that cracks it.

However public key crypto (PGP etc) was invented to solve the problems with 1-time pad exchange.

 

[Paul Uszak]

...if this is a research project,

Call it a hobby

 

[rootone]

Whatever language your application is written in, it should have some way to access libraries written in assembly language.
At assembly code level you definitely are able to access any element of memory or hard storage,and frick around with it any way you want, and you don't have to be doing it 11 times to be sure it worked.
(you might have to code it 11 times though (because assembly is a pain and often does not do what you expected).

 

[Vanadium 50]

At assembly code level you definitely are able to access any element of memory or hard storage

As pointed out before, this is not true. An SSD will present different areas of physical storage to the CPU as the same area. This remapping is done independent of the CPU, so there is nothing that can be done at the CPU level to mitigate this. (Hard drives do this as well, but to a lesser extent)

 

[rootone]

I haven't actually tried accessing an SSD directly at machine code level,so I'll accept your point if that's how they work.
However if it's the case that 'deleted' items in SSD memory, are not really deleted but just marked as 'reusable', and also that there is no way for the CPU to programatically access the physical memory directly, isn't this very nearly just as good as it being actually physically deleted?
Reason being that anyone who wanted to examine an SSD, perhaps for forensic purposes, they also would have no way to access specific physical parts of the memory which may contain something of interest to them.

 

[Carno Raar]

I haven't actually tried accessing an SSD directly at machine code level,so I'll accept your point if that's how they work.
However if it's the case that 'deleted' items in SSD memory, are not really deleted but just marked as 'reusable', and also that there is no way for the CPU to programatically access the physical memory directly, isn't this very nearly just as good as it being actually physically deleted?
Reason being that anyone who wanted to examine an SSD, perhaps for forensic purposes, they also would have no way to access specific physical parts of the memory which may contain something of interest to them.

SSDs have wear-levelling. How I think that works is that they remap the drive so that the address you wanted deleted is remapped to a new (free) part of the disk, and the part of the disk you used to have your data on is marked invalid (not free). Only as the disk gets full does it recycle (mark as free) the invalid parts. This prevents anyone area of the disk from being over-used as the memory cells have only a limited number of read/write accesses before they fail, so they have spread 'wear' evenly over the whole disk. Hence file-level scrubbing won't work. (When it reports free space, it needs to include "invalid")

If we're talking forensics, a forensics lab could take the disk apart and use standard electronics tools to examine the flash memory. You couldn't do that on a rotary drive as the data would actually be over-written. (There are some edge cases I am aware of with ext4 and non-standard disk mounting parameters).

Whatever language your application is written in, it should have some way to access libraries written in assembly language. At assembly code level you definitely are able to access any element of memory or hard storage,and frick around with it any way you want, and you don't have to be doing it 11 times to be sure it worked.

You're still limited to what the OS gives you. If you try to modify kernel memory, you'll get core dump (protection fault on Windows). You only have full access if you're actually on the metal, and not in the OS's virtual environment (OS metal > metal). Sounds pedantic, but there are OS-level logging tools that you can't bypass without admin access or an exploit.

 

[rootone]

Sure, an OS can (and so it should!), prevent somebody from attempting to meddle with storage which it has deemed protected.
I think though that the OP really meant secure deletion of just regular files using Java.
I was suggesting that one way to do it could be to provide Java with an external library designed for that purpose.

(This will not solve the problem discussed earlier regarding SSDs though.
It seems to be the case that data on an SSD cannot be guaranteed to be overwritten, (regardless of what language you are using).
It can persist in an 'invalid' state for quite a long time and while inaccessible ordinarily while in that state, it could actually be recovered by a forensic lab if need be.

 

[Carno Raar]

Sure, an OS can (and so it should!), prevent somebody from attempting to meddle with storage which it has deemed protected. I think though that the OP really meant secure deletion of just regular files using Java. I was suggesting that one way to do it could be to provide Java with an external library designed for that purpose.

You still need to talk to/patch the OS. You would have to secure erase the swap file / page file just in case the OS decided to swap your file or decryption key to disk (which is one of the reasons why I would recommend full disk encryption for this). I've not looked at the source code for PGP but I am fairly sure it must have some kind of OS-level memory lock to prevent paging. If you did that without collaboration with the OS, you'd be asking for a system crash. Obviously you can do this in an external library, but the point is that it would probably be on the same level as the OS itself.

 

[Superposed_Cat]

Yes, very easily
step 1. open file stream with pointer to location.
step 2. spam write random values for that file.
step 3. delete file
step 4. repeat steps 2 and 3,
I have done this in C# before, found this on codeproject it seems legit. Should be easy enough to translate into java.

public void WipeFile(string filename, int timesToWrite)
{
    try
    {
        if (File.Exists(filename))
        {
            // Set the files attributes to normal in case it's read-only.

            File.SetAttributes(filename, FileAttributes.Normal);

            // Calculate the total number of sectors in the file.
            double sectors = Math.Ceiling(new FileInfo(filename).Length/512.0);

            // Create a dummy-buffer the size of a sector.

            byte[] dummyBuffer = new byte[512];

            // Create a cryptographic Random Number Generator.
            // This is what I use to create the garbage data.

            RNGCryptoServiceProvider rng = new RNGCryptoServiceProvider();

            // Open a FileStream to the file.
            FileStream inputStream = new FileStream(filename, FileMode.Open);
            for (int currentPass = 0; currentPass < timesToWrite; currentPass++)
            {
                UpdatePassInfo(currentPass + 1, timesToWrite);

                // Go to the beginning of the stream

                inputStream.Position = 0;

                // Loop all sectors
                for (int sectorsWritten = 0; sectorsWritten < sectors; sectorsWritten++)
                {
                    UpdateSectorInfo(sectorsWritten + 1, (int) sectors);

                    // Fill the dummy-buffer with random data

                    rng.GetBytes(dummyBuffer);

                    // Write it to the stream
                    inputStream.Write(dummyBuffer, 0, dummyBuffer.Length);
                }
            }

            // Truncate the file to 0 bytes.
            // This will hide the original file-length if you try to recover the file.

            inputStream.SetLength(0);

            // Close the stream.
            inputStream.Close();

            // As an extra precaution I change the dates of the file so the
            // original dates are hidden if you try to recover the file.

            DateTime dt = new DateTime(2037, 1, 1, 0, 0, 0);
            File.SetCreationTime(filename, dt);
            File.SetLastAccessTime(filename, dt);
            File.SetLastWriteTime(filename, dt);

            // Finally, delete the file

            File.Delete(filename);

            WipeDone();
        }
    }
    catch(Exception e)
    {
        WipeError(e);
    }
}

 

[Carno Raar]

Yes, very easily
step 1. open file stream with pointer to location.
step 2. spam write random values for that file.
step 3. delete file
step 4. repeat steps 2 and 3,
I have done this in C# before, found this on codeproject it seems legit. Should be easy enough to translate into java.

If you had read the thread, you would realize why this is NOT secure.

TL;DR: On an SSD you have no guarantee that you are over-writing the disk block that you think you are. By design, the controller in an SSD does not always map the same disk address to the same memory circuit. There are some other minor issues like inodes on ext filesystems.

The rest of it is stuff related to the OP's specfic problem; in his case he also has to make sure the OS does not swap/page the data to disk as that would create a copy of it in the swap/page file.

 

[Superposed_Cat]

If you had read the thread, you would realize why this is NOT secure.

TL;DR: On an SSD

Sorry my post was for HDD's