ZFS on GELI kernel panic when unplugged HDD

belon_cfy · Oct 18, 2013

Hi,

Is there any bug on GELI because unplugging the disk will trigger kernel panic? No problem on a non GELI pool on the same server.

I'm able to reproduce the issue almost everytime with the following step:
- run the following command:

Code:

sh
for i in 1 2 3 4 5 6 7 8 9 10; do dd if=/dev/zero of=/vol/test$i bs=1M;done

- physically unplug the gpt/slog_disk0.eli after 10 minutes later, it will trigger kernel panic and server will be rebooted automatically.

My server specification as below:

Supermicro 1U server.
Xeon 1230v2 with AESNI supported and enabled.
8GB RAM.
FreeBSD 9.1-p7 server setup with GELI enabled.

Pool detail as below:

Code:

pool: vol
 state: ONLINE
  scan: none requested
config:

        NAME                    STATE     READ WRITE CKSUM
        vol                     ONLINE       0     0     0
          mirror-0              ONLINE       0     0     0
            gpt/data_disk0.eli  ONLINE       0     0     0
            gpt/data_disk1.eli  ONLINE       0     0     0
          mirror-1              ONLINE       0     0     0
            gpt/data_disk2.eli  ONLINE       0     0     0
            gpt/data_disk3.eli  ONLINE       0     0     0
        logs
          gpt/slog_disk0.eli    ONLINE       0     0     0
        cache
          gpt/l2arc_disk0.eli   ONLINE       0     0     0


NAME  PROPERTY     VALUE     SOURCE
vol   sync         always    local
vol   compression  off       local
vol   recordsize   128K      default
vol   atime        on        default

Below is my /etc/rc.conf

Code:

hostname="storage.ne.com.sg"
ifconfig_em0="inet 10.8.8.42 netmask 255.255.255.0 up"
ifconfig_em1="inet 10.9.9.42 netmask 255.255.255.0 mtu 9000 up"
defaultrouter="10.8.8.1"

sshd_enable="YES"
zfs_enable="YES"

geli_devices="gpt/data_disk0 gpt/data_disk1 gpt/data_disk2 gpt/data_disk3 gpt/l2arc_disk0 gpt/slog_disk0"
geli_swap_flags="-e AES-CBC -l 256 -s 4096 -d"
geli_default_flags="-p -k /boot/keys/disks.key"

# SendMail
sendmail_enable="NONE"
sendmail_msp_queue_enable="NO"
sendmail_outbound_enable="NO"
sendmail_submit_enable="NO"

# NFS
nfsv4_server_enable="YES"
rpcbind_enable="YES"
nfs_server_enable="YES"
nfs_server_flags="-u -t -n 64"
mountd_flags="-r"
rpc_lockd_enable="YES"
rpc_statd_enable="YES"

/boot/loader.conf

Code:

zfs_load="YES"
aesni_load="YES"
vfs.zfs.cache_flush_disable=1
vfs.root.mountfrom="zfs:zroot"

vfs.zfs.arc_max="4G"
vfs.zfs.write_limit_override="2G"

mav@ · Oct 20, 2013

It is difficult to identify problem without you providing any information about the panic. But I can say that month ago I've rewritten GELI destruction process and merged it to 9-STABLE. It may be related or may be not.

belon_cfy · Oct 22, 2013

mav@ said:
It is difficult to identify problem without you providing any information about the panic. But I can say that month ago I've rewritten GELI destruction process and merged it to 9-STABLE. It may be related or may be not.

Hi @mav@, seems what you have commented might be right because the same issue does not happen on FreeBSD 9.2 anymore, just simulated the test on the same server.

However due to some memory leak issue in FreeBSD 9.2, I will stick with FreeBSD 9.1 for a while which might be potentially auto reboot when HDD fail.

Sebulon · Oct 22, 2013

belon_cfy said:
However due to some memory leak issue in FreeBSD 9.2, I will stick with FreeBSD 9.1 for a while which might be potentially auto reboot when HDD fail.

I have three systems upgraded to 9.2-RELEASE in production right now and hearing about memory-leak issues is never a good thing. Would you mind elaborating that a bit please?

/Sebulon

belon_cfy · Oct 22, 2013

Sebulon said:
I have three systems upgraded to 9.2-RELEASE in production right now and hearing about memory-leak issues is never a good thing. Would you mind elaborating that a bit please?

/Sebulon

Hi @Sebulon, I'm not sure whether can it be classified as a memory leak or not, however my server was slowly losing the available memory. It didn't happen in FreeBSD 9.1 previously.
http://forums.freebsd.org/showthread.php?t=41880

My server was eventually down after 10 days of uptime.