Kernel Panic at 3:00 am

Another suggestion: have you tried monitoring your RAM usage? Do you set your ARC max (vfs.zfs.arc_max)? If not, you should. I have had major stability problems when arc_max is not set. The ARC grows too big and the kernel memory management doesn't know abnout it. The ARC will grab RAM as soon as it is available, whict other processes will lock until there is free RAM. Because the virtual memory subsystem isn't aware of ARC, even swap will not help you.

So, it could be the find command (from periodic) is triggering the ARC to grow.

You must must must set the vfs.zfs.arc_max in your /boot/loader.conf.

e.g.: on my 32GB machine (running 10 or so VirtualBox VMs):

Code: 
$ cat /boot/loader.conf                                                                                                                                                                                                                                    
## Filesystem Support                                                                                                                                                                                                                                                          
zfs_load="YES"                                                                                                                                                                                                                                                                 
libiconv_load="YES"                                                                                                                                                                                                                                                            
libmchain_load="YES"                                                                                                                                                                                                                                                           
cd9660_iconv_load="YES"                                                                                                                                                                                                                                                        
msdosfs_iconv_load="YES"                                                                                                                                                                                                                                                       
ntfs_load="YES"                                                                                                                                                                                                                                                                
ntfs_iconv_load="YES"                                                                                                                                                                                                                                                          
udf_load="YES"                                                                                                                                                                                                                                                                 
udf_iconv_load="YES"                                                                                                                                                                                                                                                           
fuse_load="YES"                                                                                                                                                                                                                                                                
                                                                                                                                                                                                                                                                               
## VirtualBox Support                                                                                                                                                                                                                                                          
vboxdrv_load="YES"                                                                                                                                                                                                                                                             
                                                                                                                                                                                                                                                                               
# Limit ARC to 12GB of RAM                                                                                                                                                                                                                                                     
vfs.zfs.arc_max="12G"                                                                                                                                                                                                                                                          
                                                                                                                                                                                                                                                                               
# sys-v style shared memory tuning                                                                                                                                                                                                                                             
kern.ipc.shmall=524288                                                                                                                                                                                                                                                         
kern.ipc.shmseg=512                                                                                                                                                                                                                                                            
kern.ipc.shmmni=384                                                                                                                                                                                                                                                            
kern.ipc.semmni=256                                                                                                                                                                                                                                                            
kern.ipc.semmns=512                                                                                                                                                                                                                                                            
kern.ipc.semmnu=256
 
ethoms said:
Sorry, I haven't had time to read the whole thread. But I had a server reboot quite often at exactly 3:00am. Basically it was a daily periodic script that checked mount points. On this particular server I had an sshfs remote filesystem mounted. The security script, ran at 3:00am daily would cause a panic (I guess).
Click to expand...

Sounds like you found a bug with either the FUSE implementation or the SSHFS portion of it. If it's reproducible, you should probably look around on Bugzilla for anything like it and open a ticket with the details if you can't find anything. I remember hear the FUSE implementation had some issues so that may help in nailing them all down.
 
Well it's hard to reproduce now. I've re-installed the server with a much newer version of FreeBSD. I think I disable that option in /etc/periodic just in case, or as well as I don't ned the security check. Anyway, the FUSE has been replaced since then by a better implementation. I have much better experience with the new FUSE support.
 
Earlier in this discussion I posted about this. That was an older laptop I was tinkering with, and turns out it was the hard drive that finally failed. Thanks for your help with that.

So now, one of my new-to-me servers is doing this. An I can reproduce it manually at will by running the find command. I am suspicious that find alone is not the culprit, but may have to do with another process that is running at the same time. The interesting thing is that I have two freshly built servers that are very similar, and one is fine so far. Both are FreeBSD 10.1-RELEASE on Dell Dimension boxes. The working one has a 40GB OS drive, 2TB content/nas drive, and 4GM ram. The failing one has 140GB OS drive, and 2TB content/nas drive, and 1GB ram. Both have UFS files systems.

There are no crash dumps or reboots when the fault occurs. I just get an OS freeze. The server is not pingable, and I cannot console into it. Tapping the power button does not cause a shutdown. I have no choice but to power cycle. I do wonder if its because it only has 1GB of ram, but it does have 4GB of swap and during the freeze only 24M was used. I cannot find any logs of any sort, but maybe I am not looking in all the right places.

If anyone has any thoughts, please post, otherwise stay tuned. I'll see what I can figure out.
 
Well, in this thread I had almost all five HDDs faulty using ZFS, that's why find panic'ed every time it started through /etc/periodic.conf.
 
So running find many many many times by itself no issues. Running it with another program using significant cpu resources, and I assume disk resources, causes the OS to freeze. The disk being used though is a new one, not an old one. Gonna see if I can figure out what exactly is happening, and let you know. I'm thinking now that the other program is causing the freeze.
 
You must log in or register to reply here.
Back
Top