ARMv7 filesystem corruption after power loss

E

eldaemon

On Beaglebone white and Raspberry Pi 2B:

pkg install e2fsprogs

sync

Pull power.

Power on.

badblocks: command not found (no file there)

pkg info shows e2fsprogs is installed. pkg remove e2fsprogs complains about missing files.

pkg install e2fsprogs, then badblocks complains about missing libintl.so.8.

I'm a bit confused about this, assumed journaling would protect against it. Even so, I ran sync first so journaling shouldn't even be needed. Two different devices, two different SD cards. FreeBSD 12.1, official releases.

Anyone know what might be going on?
 
Looking at the man page for tunefs is states that the file system must be in read only mode (single user) or unmounted...
 
Thank you. You're right.

Wasn't able to mount -o ro / live, had to `shutdown now` then `tunefs -n disable /`.

Rebooted to softupdates being disabled.

Did the same test `pkg install e2fsprogs; sync` and pulling the power.

badblocks works! It was softupdates after all.

Now I'm curious to test how long I need to wait before pulling the power to avoid corruption.
 
Yeah, I guess this all works as intended. I just wish I knew how to force a sync through the softupdates layer. I guess the shutdown scripts might give me a hint there.

Seems kind of unintuitive to be able to `sync` and have data loss like that.
 
Pulling the power is never a good idea, regardless of the OS. Always use a proper shutdown; shutdown -p now.

One way to prevent corruption is to make everything read-only. Use a tmpfs(5) for things like /tmp and /var/tmp and send logging to a remote server.
 
Right, I know it's not a good practice to pull the power. But it happens and it's good to know what happens and what level of consistency you can expect.

In this case I found out by accident. The microUSB cable I have going into my Pi is extremely sensitive, a small bump caused it to reboot and I lost my package. Repeated testing showed the same thing.

If I'm running servers I want to know what happens if there's a power outage (and UPSs run out, etc) that I can't shut them down gracefully for.
 
eldaemon said:
If I'm running servers I want to know what happens if there's a power outage (and UPSs run out, etc) that I can't shut them down gracefully for.
Click to expand...
If you have UPSs then you need to configure it so it cleanly shuts down systems when the battery is starting to get critically low.

Just pulling the plug (for whatever reason) is like playing Russian roulette. There's a 5:6 chance nothing happens, it's the 1:6 that will kill you. The more often you pull that trigger the higher the likelihood is that you die.
 
I used to have a powerbank on my raspberry pi 3B. Unfortunately, those tinkerboards cannot measure the incoming voltage on the microusb port, and thus cannot calculate any remaining battery life. ;D It might be impossible to cleanly shutdown in time.
 
Aren't USBs supposed to have 5V regardless of remaining charge? Li-ion battery (single cell) is around 3.7V-4.x V but there is probably power regulator to convert that to expected 5V.

You can calculate remaining battery capacity with something like I2C battery gauge IC (with disassembled battery, of course). Unfortunately, my ARM board doesn't have working I2C so I have never tested it.
 
There are two sources of data loss when cutting power. One is the OS itself, which may hold writes in buffer cache before staging them out to the hardware devices. That area is very well understood. And in your case, you ran "sync", so everything should have gone to the devices.

The second part of the system that can lose data are the devices themselves. In the old days, those were hard disks (spinning rust). They are actually very very good about writing everything that remains in their internal cache to disk even on power loss; disks use the remaining rotational energy of the platter to power the electronics (using the spindle motor as a generator, known as back-EMF) and finish the last few sector writes. But this doesn't work for flash storage, for lack of energy storage. Regular SSDs (the ones in physical aluminum boxes with SATA or SAS connectors) have capacitors, but even those are often not big enough to finish the last few writes. One of the reasons is that writing to flash uses a lot of energy, considerably more than reading. Some SSDs use DRAM as a buffer cache, and protect the DRAM with supercaps (on shutdown they don't even try to write to flash, to most efficiently use the remaining energy in the capacitor). That's all great, but for small formfactor storage devices (such as SD card, CF, and M.2), there is no room for physical capacitors. And on a physically small system (like BeagleBone or Raspberry Pi) there are very few capacitors on the "motherboard", so power just goes away near instantaneously, and pending writes get lost. This is sort of unavoidable: we're using media that was really intended for digital cameras and cell phones (which all have batteries that are well monitored, and we know when they are about to shut down) in computers with no shutdown protection, and the result is data loss and corruption.

On one of my Raspberry Pi, I use a "tiny little UPS": This little board from Adafruit is a power splitter that can be powered either from USB or a Li-Po battery, and then I plug a physically small LiPo battery into it. This is not for the two RPi that are in production (they are protected by battery-derived power, and have reasonably wiring), but for the one that's on my workbench. The reason is that on a workbench you are always messing around, touching connectors, pulling cables, and too often I crash the RPi. And I was working on connecting a cell phone modem to my RPi, so it was handy to be able to unplug it from a power outlet, and wander around the house to see how reception changes.
 
aragats said:
Slightly off-topic: It doesn't make much sense to run badblocks on SD/MMC, since they have multiple layers of embedded software/hardware, and you're not going to test single blocks (or even test anything at all).
Click to expand...

I'm running badblocks on a USB harddrive enclosure.
 
I too have a similar problem with 3 Raspberry Pi's running 12.1 after losing power on the devices owing to a area-wide power blackout. There were 2 x R-Pi_B and 1 x R-Pi_2. The mmc/sd cards are all ok and fully operational but now I have 4 USB sticks marked as read-only and neither FreeBSD 12.1, Linux Mint 18.3, Linux Mint Debian Edition 4 or Windows 7 can remove the read-only flag marker. Any help is appreciated as the USB sticks are now virtually useless as they cannot be formatted or written to.
 
Try just erasing them using dd(1), just overwrite everything with zeros. Regardless of how corrupt the filesystem may have gotten you should still be able to write to the raw disk.
 
tarahall149 said:
I too have a similar problem with 3 Raspberry Pi's running 12.1 after losing power on the devices owing to a area-wide power blackout. There were 2 x R-Pi_B and 1 x R-Pi_2. The mmc/sd cards are all ok and fully operational but now I have 4 USB sticks marked as read-only and neither FreeBSD 12.1, Linux Mint 18.3, Linux Mint Debian Edition 4 or Windows 7 can remove the read-only flag marker. Any help is appreciated as the USB sticks are now virtually useless as they cannot be formatted or written to.
Click to expand...
What's the file system?
 
Alain De Vos said:
I don't know the booting mechanism, but can't you boot in single user mode with ro mount of / and perform a fsck /
Click to expand...
The SD card(s) that they boot from are OK and fsck works OK on them, the USB drive just supplies additional storage.
I had "./ports" directory located on the USB using a symbolic link from /usr on the SD card in order to save space on that medium.
Using bsdconfig, the USB drive is configured as:

da0=MBR, da0s1=BSD, da0s1a=BSD, freebsd-ufs, mount point=/.

When try to mount it the message is: "read-only filesystem"
If I try deleting it I get: "read-only filesystem, geom, da0s1".

Using: dd if-dev/zero of=/dev/da0s1a returns "read-only filesystem"
-----------------------------------------------
Windows 7 diskpart.exe returns:

Current Read-Only State=YES
Read Only=NO
Boot Disk=NO
Pagefile Disk=NO
Hibernation Disk=NO
Crashdump=NO
Clustered Disk=NO
------------------------------------------------
Linux GPartED will delete the partition (so it says) but wont allow to make a new partition and returns:
"Input/output error during write on /dev/sdb"
"Can't write to /dev/sdb, because it is opened read-only"

If I cannot clean that read-only flag then I have lost 4 almost new USB drives, 2 x 16gb & 2 x 32gb.
I say almost cause they were new when I started. To say I would like to sue the electricity provider would be understating what I would really like to do.
 
Sounds like maybe they are terrible quality USB sticks that went into some read only mode after they detected corruption. Assuming they don't have physical write protect switches which are set.

There are lots of fake/garbage USB sticks out there, like the kind that say 1TB for $20.
 
You must log in or register to reply here.
Back
Top