Shell Fast and easy delete of partition and filesystem data

[Sebulon]

I've also posted this in another thread but thought it be best to start a new post about it since it is more of a general purpose tool. Been tinkering on this for a bit and I am now ready to present to everyone:

Latest version:

#!/bin/sh

if [ -z "$1" ]
  then
    echo "Usage: $(basename $0) drive1 drive2 ..."
  exit
fi

SMARTCTL=$(whereis smartctl | awk '{print$2}')

if [ "$(echo ${SMARTCTL} | wc -w)" -le "0" ]; then
  echo "Error! \"smartctl\" command was not found!"
  echo "Please install the \"smartmontools\" package and try again!"
  exit
fi

verifydrives() {
  for HDD in $HDDS; do
    if [ "$(ls -1 /dev/ | egrep -c "^${HDD}$")" -le "0" ]; then
      echo "Hard drive \"${HDD}\" does not exist. Aborting."
      exit
    else
      echo "Hard drive \"${HDD}\" verified."
    fi
  done
}

cleandrives() {
  for HDD in ${HDDS}; do
    STARTPOINT=$(${SMARTCTL} -i /dev/${HDD} | \
                 awk '($1=="User" && $2=="Capacity:")\
                 {{for(i=3;i<=NF;++i)if($i~/^[0-9]+/)\
                 var=var$i};gsub(/\xa0/,"\x00",var);\
                 print (var / 1024) - 10}')
    dd if=/dev/zero of=/dev/${HDD} bs=1M count=10 >/dev/null 2>&1
    dd if=/dev/zero of=/dev/${HDD} bs=1M count=10 seek=${STARTPOINT} \
    >/dev/null 2>&1
  done
}

HDDS="$*"

verifydrives ${HDDS}

echo ""
echo "This will irreversibly destroy partition- and filesystem data on drive(s):"
echo "${HDDS}"
echo ""
echo "USE WITH EXTREME CAUTION!"
read -r -p 'Do you confirm "yes/no": ' CHOICE
case "${CHOICE}" in
  yes) cleandrives ${HDDS}
       echo ""
       echo "Drive(s) cleaned."  ;;
   no) echo ""
       echo "Cleaning cancelled."; break ;;
    *) echo ""
       echo "Cleaning cancelled."; break ;;
esac

This script verifies and then erases the first and last 10MB of every hard drive you´ve confirmed. If you find yourself in a situation where you don't want to wait for a complete dd over a 1 or 2 or even 3TB drive, or if you want to move a hard drive from one system to another and don´t want the new system to be "confused" by old partitioning- and filesystem data, then this command makes that cleaning fast and easy. Best part is that it works on any *nix system out there, not just FreeBSD. Enjoy!

/Sebulon

 

[nekoexmachina]

        if [ `ls -l /dev/ | grep $drive | wc -l` = "0" ]
          then
            echo "Drive $drive does not exist. Aborting."
            exit
          else
            echo "Drive $drive verified."
        fi

Better check if it exist with file /dev/$drive
or if [ -x /dev/$drive]
Cause grep will say yes, it is if i will run your script like $ script.sh da if there will be da0 or da1 or something.

 

[wblock@]

Some general thoughts...

Automating this makes me a little queasy. The code should be very, very picky. Easy to safely quit and hard to actually wipe a disk. Showing the user information about the device they're about to destroy, like with file -s, could be useful.

Along those lines, a more suggestive confirm choice would help alert the user they're about to do something serious. Maybe "destroy disks" instead of "yes". "no" isn't needed, it's the default choice.

To erase MBR and/or GPT, the first and last 35 blocks (17K) of the drive is enough. The first 35 blocks alone is probably adequate.

bs=1m breaks on Linux dd(1), which is case-sensitive on block size (M for meg, K for k). A buffer that large really isn't needed, 64K is plenty.

The GEOM safety will prevent writes to important parts of the disk like the MBR. It can be disabled by setting sysctl kern.geom.debugflags=16.

 

[Sebulon]

Thank you both for your inputs, they are now corrected.

if [ -x /dev/$drive]

Wow, that easy huh? I´m trying to remember if I didn´t try it just like that but it didn´t work and stopped and kept saying "not permitted" something or other...and that´s why I went that other way instead.

Would any of you happen to know if the default block-size of dd is the same everywhere; 512b? Cause it´s kinda important for this command to be accurate on other platforms as well. If not, any ideas on how to improve?

Just nuking the beginning of a drive has been unsuccessful for me, at least with ZFS, it can still find the backups at the end. I have also read from at least three different people on this forum that it´s safest to wipe 10MB for good measure. Just for partition data, fine, but this aims to wipe filesystem data as well, so that´s why it is that way.

It´s not this tool´s job to know whether you have set the geom.debugflags to 16. Everyone should know to do that beforehand.
Besides, it´s very FreeBSD-specific. Also:

sysctl kern.geom.debugflags=16 is being used only if the disk is currently mounted in read/write mode. Something just to protect you from accidentally messing up.

If the drives you doing this on are mounted read/write, you are doing something wrong

/Sebulon

 

[Sebulon]

Learned something new about dd today. If you seek and have specified bs, it will use that unit to seek with. It came to me while trying to use bs=1M, while seeking to the end of the drive.

Writing 10MB with bs=1M took 0.05 secs
Writing 10MB with bs=512B took 7.9 secs

So I wanted to change the second dd to use bs=1M, but it refused, because I used the total number of blocks to seek with. So I had work out a way to use dmesg, and retrieve the size in MB, minus 10. And it had to work cross-platform. Well, now it does I have tested this on both FreeBSD and Ubuntu and it works.

Total time of cleaning one drive: 0.1 second.

/Sebulon

 

[priyadarshan]

The GEOM safety will prevent writes to important parts of the disk like the MBR. It can be disabled by setting sysctl kern.geom.debugflags=16

I know, this is a really old thread, but wblock@ reference to that sysctl saved my sanity.

I repeatedly zerod'ed an SSD drive, and, once FrreBSD was reinstalled, I was still able to see all old content.

For many hours I though it was actually booting from another similar drive, which happened to have same OS.

But, at one point I realised uname -a was returning a different result. I could not believe, all the information was still there!

The sysctl kern.geom.debugflags=16 incantation seemed to do the trick.

----

Update: Actually, the issue is deeper than I thought. I am not sure at what point zeroi'ing was taking effect (before or after incantation).

But, no matter what I do, the newly zero'ed and freshly installed SSD as ndv0, at boot time will show the content of nvd1.

This is such a mistery to me.

I wil bring this to a new thread. Sorry for the noise.

 

[Sebulon]

I thought it´d be fun to update the script to see if there where things that could be done more "elegant" than in my first version, so I´m keeping the old version underneath, mostly for my own amusement, but also to explain some subtle differences, for future reference.

First thing I wanted to change was how I made sure that a hard drive existed. @nekoexmachina had the great idea to use the builtin shell test utility. But for some reason all the hard drives in my NAS are character special devices, instead of the block devices they are treated as in Linux:

FreeBSD:

$ for i in c b; do test -$i /dev/da0; echo $?; done
0
1

vs. Linux:

$ for i in c b; do test -$i /dev/sda; echo $?; done
1
0

So I wanted to have a uniform way of doing it and came to the conclusion that my initial solution, while being too opportunistic, was the most portable way, but strengthen the way it was evaluated.

Next thing I wanted to change was how to get size of the hard drives in question. My version used grep to go through dmesg to find that data, which works perfectly well in FreeBSD (and possibly other BSD's) but not in Linux, because they don´t keep any dmesg.boot file lying around, forcing me to work with what´s reported by running the dmesg command. That, as you may know, is no good since boot information quickly rotates away and so won´t contain the information you need. fdisk exists but doesn´t work in the same way as in Linux and gpart is non-existent in Linux. The only tool I could think of, that I ended up using is smartctl, which in my opinion is as drag, because it adds this as a dependency, but who doesn´t have smartmontools installed in a NAS anyway, right? If you can think of anything else that is completely the same, please let me know! Why does it have to be the same, you wonder? Well, because I´m lazy, that´s why

I also wanted to cut down the number of lines used in the script and was able to achieve that by removing an entire function that was essentially replaced with one awk command God, I love awk! It can be a real b**** to wrap your head around. I, myself, am having most trouble understanding it´s syntax (especially inline) and it could do a lot more helping out with the debugging, but I cannot think of a more versatile tool, it really can do everything you want.

E.g. how I noticed that just printing the output and trying to do math on it with like expr or bc would yield errors. expr said the numbers wheren´t really numbers and bc told me it was because there had snuk in some wierd ASCII characters in it. Immediately, I started writing a pipe output to sed to get rid of them, but then I thought "Why am I doing this, I have everything I need right here!", so I wrote a gsub statement in awk to get rid of them. Done!

Now my calculations worked just fine in expr and bc, when I had the same feeling all over again. Of course I didn´t have to pipe anything when I was able to do the math inside of the same command! Last but not least, I looked back and saw that I ran smartctl, pipe'd to grep to look for "User Capacity:", which is where the size is printed, and then handed over to awk to do the rest, when I thought: "God, how stupid I am!", I could´ve done it all with just awk all the way!

So in conslusion, I just wanted to say that something that could´ve have been done with grep, cut, tr, probably sed and bc (or expr) can be done with just "the one" 5:1 is a damn good ratio!

All this, together with some style changes have lead to a much more elegant tool, in my humble opinion

/Sebulon

Original/Old version:

#!/bin/sh

if [ -z "$1" ]
  then
    echo "Usage: `basename $0` drive1 drive2 ..."
  exit
fi

drives="$*"

verifydrives()
  {
    for drive in $drives
      do
        if [ `ls -l /dev/ | grep -w $drive | wc -l` = "0" ]
          then
            echo "Drive $drive does not exist. Aborting."
            exit
          else
            echo "Drive $drive verified."
        fi
      done
  }

seeksector()
  {
    blocksize=`dmesg | grep -w $drive | grep -oe '[0-9]\{7,\}'`
    mbsize=`echo "$blocksize / 2048" | bc`
    echo "$mbsize - 10" | bc
  }

cleandrives()
  {
    for drive in $drives
      do
        dd if=/dev/zero of=/dev/$drive bs=1M count=10 >/dev/null 2>&1
        dd if=/dev/zero of=/dev/$drive bs=1M count=10 seek=`seeksector $drive` >/dev/null 2>&1
    done
  }

verifydrives $drives

echo ""
echo "This will irreversibly destroy partition- and filesystem data on drive(s):"
echo "$drives"
echo ""
echo "USE WITH EXTREME CAUTION!"
read -r -p 'Do you confirm "yes/no": ' choice
  case "$choice" in
    yes) cleandrives $drives
         echo ""
         echo "Drive(s) cleaned."  ;;
     no) echo ""
         echo "Cleaning cancelled."; break ;;
      *) echo ""
         echo "Cleaning cancelled."; break ;;
  esac

 

[ShelLuser]

I'm still not much of a fan on automating this but I do think it's an improvement that you no longer rely on smartd, it's really not as reliable as it may seem.

Still, if you want to get the currently available storage media in a reliable way just use the kernel: sysctl kern.disks. You can easily filter out the optical media because that starts with cd so all that's left is to grab the rest. Even relying on /dev entries isn't the best solution here because it's also not that hard to meddle with devd.

 

[Sebulon]

I'm still not much of a fan on automating this but I do think it's an improvement that you no longer rely on smartd, it's really not as reliable as it may seem.

I think you are misunderstanding, it´s actually the other way around, I´m even crazier than you think

The original, old version is in my post directly above yours. The updated, new version is in the top post! I did it like that so that people that come here looking for it don´t have to read the entire thread to find the newest version available.

$ sysctl kern.disks
sysctl: cannot stat /proc/sys/kern/disks: No such file or directory

Unfortunately doesn´t work in Linux. Besides, I want the size of them, not what they are called. Thanks for the tip though!

/Sebulon

 

[SirDice]

But for some reason all the hard drives in my NAS are character special devices, instead of the block devices they are treated as in Linux

This can be explained: FreeBSD Architecture Handbook: 9.4. Block Devices (Are Gone)

 

[waywardnl]

I agree with, wiping drives are really easy, so I adjusted the script a bit for my needs.

#!/bin/sh

if [ -z "$1" ]
  then
    echo "Usage: $(basename $0) drive1 drive2 ..."
  exit
fi
if [ -n "$2" ]
  then
    echo "Only Erase one drive at a time!"
  exit
fi

SMARTCTL=$(whereis smartctl | awk '{print$2}')

if [ "$(echo ${SMARTCTL} | wc -w)" -le "0" ]; then
  echo "Error! \"smartctl\" command was not found!"
  echo "Please install the \"smartmontools\" package and try again!"
  exit
fi

verifydrives() {
  for HDD in $HDDS; do
    if [ "$(ls -1 /dev/ | egrep -c "^${HDD}$")" -le "0" ]; then
      echo "Hard drive \"${HDD}\" does not exist. Aborting."
      exit
    else
      echo "Hard drive \"${HDD}\" verified."
    fi
  done
}

cleandrives() {
  for HDD in ${HDDS}; do
    STARTPOINT=$(${SMARTCTL} -i /dev/${HDD} | \
                 awk '($1=="User" && $2=="Capacity:")\
                 {{for(i=3;i<=NF;++i)if($i~/^[0-9]+/)\
                 var=var$i};gsub(/\xa0/,"\x00",var);\
                 print (var / 1024) - 10}')
    dd if=/dev/zero of=/dev/${HDD} bs=1M count=10 >/dev/null 2>&1
    dd if=/dev/zero of=/dev/${HDD} bs=1M count=10 seek=${STARTPOINT} \
    >/dev/null 2>&1
  done
}

HDDS="$*"

verifydrives ${HDDS}

echo ""
echo "This will irreversibly destroy partition- and filesystem data on drive(s):"
echo "${HDDS}"
echo ""
diskinfo -v ${HDDS}

echo "USE WITH EXTREME CAUTION!"
read -r -p 'Do you confirm "yes/no": ' CHOICE
case "${CHOICE}" in
  yes) cleandrives ${HDDS}
       echo ""
       echo "Drive(s) cleaned."  ;;
   no) echo ""
       echo "Cleaning cancelled."; break ;;
    *) echo ""
       echo "Cleaning cancelled."; break ;;
esac

 

[Sebulon]

I agree with, wiping drives are really easy, so I adjusted the script a bit for my needs.

Thank you! Your addition limited the execution to only one disk at a time, which is understandable, but I have had to clean drives in bigger chassis, like 24 drives and up, and in those cases, just seeing the drives blink one after another is so satisfying, it's like the closest to storage pr0n you can get ?

 

[priyadarshan]

Thanks for your script, I have been happily using it since many years. Today I saw that some NTFS disk were still carrying some labels ,

# lsblk
DEVICE         MAJ:MIN SIZE TYPE                                          LABEL MOUNT
ada0             0:118  15T GPT                                               - -
  ada0p1         0:188  16M ms-reserved        gpt/Microsoft%20reserved%20partition -
  ada0p2         1:6    15T ms-basic-data          gpt/Basic%20data%20partition -
  <FREE>         -:-   1.0M -                                                 - -

- -



Nothing that gpart destroy -F /dev/ada0 would not fix.

 

[Cath O'Deray]

… This script verifies and then erases the first and last 10MB …

… at least with ZFS, … the end. … 10MB for good measure. …

Smart.

… four copies of the label on each device. …

More specifically, from Sun's draft ZFS On-Disk Specification (2006-08-22):



Sebulon you might enhance your script by conditionally preceding use of dd(1) with use of zpool labelclear .

<https://openzfs.github.io/openzfs-docs/man/8/zpool-labelclear.8.html>

Rationale: in an edge case, failure to overwrite (with dd) one or more of the four might leave the device in a state with the potential to cause trouble.

Whether zpool will fail more gracefully if hardware (i.e. media) is the cause of failure, I don't know (sorry), but it's food for thought.