Solved Best way to verify disk is zeroed?

`Orum

Active Member

Reaction score: 24
Messages: 199

I've written a script to wipe all disks on any machine that netboots. It works fairly well, but I'd like to add verification of the final "zeroing" pass, and only shutdown the machines if the drive reads all zeros successfully, and otherwise show the error. I'm also trying to both minimize dependencies on ports if possible (currently only using pv(1) to show status), and make the final "read" pass perform quickly (the write passes are already fast).

However, I can't figure out what the best way to do this is. It seems the easiest way is to use another port, security/bcwipe, and run it with bcwipe -bfmz /dev/adaX, which both zeros the disks and verifies the write, but this requires another port. I've looked at other, simpler ways of doing using basic system utilities, like od(1), but then the disks read slowly (about 1/3rd of the speed dd(1) will do with bs=1m). I suspect this is because you can't specify a buffer size for od, and it's reading small chunks.

Any thoughts or suggestions?
 

ikbendeman

Well-Known Member

Reaction score: 20
Messages: 357

dd of=/someplace/temp if=/dev/disk bs=nM count=n
Set blocksize and count to your hearts desire, then you can limit and test chunks of the disk by dd'ing out to another disk... then you can simply
cat /someplace/temp | more

Of course, after you have dd'd with if=/dev/zero. You can also turn on verbosity, which should give you a decent idea of whether or not it happened.
 

ikbendeman

Well-Known Member

Reaction score: 20
Messages: 357

You could also cat out to a boolean of !0 input, throws a flag in your script.
 
OP
OP
O

`Orum

Active Member

Reaction score: 24
Messages: 199

I'm not sure how copying the entire disk to a file is going to help, nevermind the fact that it's not possible on these machines (single large disks). And cat is really intended for reading text files, I have no idea what it does when it encounters a null.

The best I've come up with so far is to use dd to actually read the disk (as you can force large chunks to be read) and pipe that to another utility. Currently I'm using dd if=/dev/adaX bs=1m | cmp /dev/zero -, as 'cmp' appears to be faster than 'od' for this purpose, and outputs more useful return codes. It's still not as fast as a verification pass from bcwipe though (only ~60% as fast), so I still am looking for better options.
 

ikbendeman

Well-Known Member

Reaction score: 20
Messages: 357

where you enter count, you don't select the whole disc, just a segment, I.E. 100KB... then you cat to see if it's 0'd, or not. How concerned, exactly, are you with data destruction? An example of what you could do is
dd if=/dev/zero of=/dev/adaX
dd if=/dev/random of/dev/adaX
dd if=/dev/zero of=/dev/adaX
dd if=/dev/adaX of=/mnt/temp bs=10k count=1
cat /mnt/temp | grep 1

if nothing greps out, then it's all zeros, but in reality, upon completion of dd command it should give number of bytes, etc... check the man.

The only verification would be to check,bitwise, i.e. the way I suggested, at random sectors... but if it's not zeroing out, you have a driver, disk, or rootkit problem that likely won't be solvable on that system, as I've yet to see a system where /dev/zero represents any non-zero bits.
 

ikbendeman

Well-Known Member

Reaction score: 20
Messages: 357

Also, if the disk is zeroed out, checking the MBR/GPT boot sectors, and verifying they are zeroed would be a good indicator, per command suggested... and all you would need is a small, writeable USB or other removable media, to copy X sectors to, and then to verify... I trust you already have alternative storage as you are running a system and zeroing disks.
 

ikbendeman

Well-Known Member

Reaction score: 20
Messages: 357

more or less what I was suggesting... you could always cat the disk and pipe into more if you are on a system that doesn't have a hex editor, seeing as how you're only running for verification..
 

rufwoof

Active Member

Reaction score: 76
Messages: 235

Isn't dd'ing with /dev/random better than /dev/zero for security purposes as with all zeros residual former values might still be determined on magnetic discs.
 

ikbendeman

Well-Known Member

Reaction score: 20
Messages: 357

Isn't dd'ing with /dev/random better than /dev/zero for security purposes as with all zeros residual former values might still be determined on magnetic discs.
That's not a question with a yes or no. In certain circumstances, depending on who's analyzing your disks, it has been known to be possible for certain operators to recover data from both physically destroyed platters and, as far as I know, data which has had up to 5 (possibly 7, uncertain) writes.

If you're worried about a common criminal, your local tech store, or local police, a single write will be enough, unless they deem your data valuable enough to send it off i.e. to one of the super expensive private data recovery services in Switzerland or another neutral country.

I recommend a couple passes with /dev/random, if you're increasingly paranoid, re-seed entropy (restart, or whatever), between passes, then pass with /dev/zero. This level of data destruction is not really always advisable as if the data is that sensitive, complete physical destruction of disks should occur. Either way, whether it's crap or not, in certain jurisdictions, data destruction techniques are easily detected and can sometimes warrant only a closer look into you.
 

ikbendeman

Well-Known Member

Reaction score: 20
Messages: 357

That video from above is the last of a few steps for the governments SOP for disc destruction
 
OP
OP
O

`Orum

Active Member

Reaction score: 24
Messages: 199

Anyway, to get back on topic, I wanted to verify the entire disk is zeroed, not just randomly scan it. Assuming cat piped to less/more would work, it's still very slow--only about 70MB/s when dd reads at ~195MB/s ( bcwipe is comparable to dd in terms of read speed when verifying).

If I can simply cat to less/more, wouldn't piping dd's output to more/less/grep work as well? At least then I could verify at a reasonable rate without having to use a port.

Edit: I forgot to mention, the only other storage is have on these machines is memory disks. These are netbooting machines as I mentioned in the original post, and the network shares are read-only.
 
OP
OP
O

`Orum

Active Member

Reaction score: 24
Messages: 199

Well I'm trying to do an unattended shutdown if (and only if) it's zeroed, that's the trick. The issue is that usually piping to anything out of dd (I've tried od, cmp, and a few others) seems to kill the speed of dd.
 

olli@

Well-Known Member
Developer

Reaction score: 375
Messages: 486

In this particular case I would write a tiny C program for this purpose. The C compiler is in the base system, so you don't need to install additional ports.
Code:
#include <stdio.h>
#include <stdlib.h>
#include <stdint.h>
#include <unistd.h>

#define BLOCKSIZE (128 * 1024)
uint64_t data[(BLOCKSIZE + 7) / 8];

int
main (void)
{
        int inbytes, i;

        while ((inbytes = read(STDIN_FILENO, data, BLOCKSIZE)) > 0)
                for (i = 0; i < (inbytes + 7) / 8; i++)
                        if (data[i])
                                exit (1);
        if (inbytes < 0) {
                perror (NULL);
                exit (2);
        }
        return 0;
}
Save that source code as “testzero.c”.
Then compile it like this: cc -O2 -o testzero testzero.c
That'll give you a binary named “testzero”. Put it somewhere in your $PATH so your shell can find it, or type “./testzero” to run it from the current directory. The program reads a file (or device) from standard input and exits with code 0 if the file is all zeros, so you can use it in a shell script like this:
Code:
if testzero < /dev/ada0; then
        echo "allright, disk is zero"
        shutdown -p now
else
        echo "something went wrong!"
        yes | tr -c x '\a'        # call attention
fi
Note that I have set the block size to 128 KB, not 1 MB. In my (limited) testing it was faster with 128 KB (actually, as fast as the speed of the physical disk). This may depend on your kernel settings, CPU cache or file system parameters, though. YMMV, so you might want to test different values. If you change it in the source code, don't forget to recompile the program.
 

ikbendeman

Well-Known Member

Reaction score: 20
Messages: 357

olli@ very clever. I would have wasted time writing a tcsh script. I didn't test it, but why calling main with void?
 

olli@

Well-Known Member
Developer

Reaction score: 375
Messages: 486

I didn't test it, but why calling main with void?
For simplicity, the program reads the file or device from standard input, so there is no reason to parse the argument vector, so I just specified it as “void” instead of the usual “int argc, char *argv[]”. Besides, the compiler issues a warning if you specify argc and argv without actually using them. Note that you could omit the formal parameter completely (i.e. “()” instead of “(void)”), but I think it's a good habit to specify “void” for documentation purposes, so the author's intention is clear. For similar reasons the return type is specified as “int” – actually that wouldn't be necessary because int is the default return type (not void!). In other words: You can just write “main ()” instead of “int main (void)” if you want – both mean exactly the same, but I adhere to the Python motto “explicit is better than implicit”.

Just in case someone wonders: There is no difference (performance-wise) between using standard input vs. opening a file specified on the command line. Just make sure that the shell opens the file directly and passes it to the program (using redirection syntax with “<”). Do not use something like “cat /dev/foo | testzero” because this will create an additional process for the cat command and connect it with a pipe to the testzero command – the pipe may reduce the performance considerably. (NB: The cat command is often abused; I guess that 99% of uses of cat in shell scripts are superfluous, sometimes even harmful. )
 
OP
OP
O

`Orum

Active Member

Reaction score: 24
Messages: 199

[ "$(dd if=/dev/adaX)" = "" ] && poweroff
This is extremely slow as it reads only one character at a time. I'm thinking the best solution might just be what olli recommended and compiling my own small program.
 

VladiBG

Aspiring Daemon

Reaction score: 266
Messages: 647

For secure erase there's no fast method as you need to write at least two passes of random data to the disk. My suggestion is first to make a write pass of random data then second pass with zero only. Then there will be no need to verify the entire disk if it contain zeros it will be enough to check three parts of the disk (start sectors / mid / end sectors) if they contain zeros.

The tools that you can use for this are:

camcontrol(8) - to get the device name and to use the sanitize function for erasing the disk using crypto data or format for low level format. <- this alone will be enough if you don't need to verify the disk deletion.
diskinfo(8) - to get mediasize in sectors
dd(1) - to read the specific sector (use seek skip and count)
hexdump(1) - dump the output of dd
md(1) - calculate the checksum of the same count of sectors and compare it to know md5 value for example MD5 ("0000000") so you can verify different parts of the disk if they contain zeros.

so combining all of above you can make some script that will get the dev name, it's size, issue the sanitize of the that disk with random data or with pattern of 0 then based of the disk size you will check the three zones of the disk with the same number of sectors and compare them to md5 checksum to verify if it's zeroed.
 

olli@

Well-Known Member
Developer

Reaction score: 375
Messages: 486

For secure erase there's no fast method as you need to write at least two passes of random data to the disk.
Actually, no. On modern hard disks the bit density is so high that a single pass is enough for all practical purposes. And it even doesn't matter if you write random data or just zeros. (Note that /dev/random is much slower than /dev/zero because of algorithmic complexity.)

And if that is not secure enough for you, then there is in fact a way that is more secure and much faster: Open the disk case with a screwdriver, remove the platters and wipe them with sandpaper. Watch your data disappear in the dust. Forget all that nonsense with magnets, hammers, knifes or anything – sandpaper is best because it's quick, safe, and it's guaranteed to not leave a single bit on the surface. While you're at it, you might also want to physically destroy the chips inside, to make sure that no data remains within cache RAM.
The only drawback of this method: You cannot reuse the drive afterwards, except as a door stop. You can also use the platters for decorative purposes. ;)
 
Top