panic: kmem_malloc : kmem_map too small

ShyRain

ShyRain

Hi,
my server reboots every night at 3:05 AM last 10 days.
During reboot no write anything to dmesg, messages and audits.

Finally, I got snapshot of last screen before panic,
deikm.jpg


What is the solution?


Crontab was empty, who is the find -sx commands owner? It works once a day at 3 AM. I checked /etc/newsyslog.conf, /etc/syslogd.conf but not included any scripts which has this command.

Code: 
#ps fauwwx
USER      PID %CPU %MEM   VSZ   RSS  TT  STAT STARTED      TIME COMMAND
root    79216  7.2  0.0  5836  1856  ??  D     3:03AM   0:04.35 find -sx / /usr /opt/myfolder /dev/null -type f ( -perm -u+x -or -perm -g+x -or -perm -o+x ) ( -perm -u+s -or -perm -g+s ) -exec ls -liTd {} +
 
ShyRain said:
Crontab was empty, who is the find -sx commands owner? It works once a day at 3 AM. I checked /etc/newsyslog.conf, /etc/syslogd.conf but not included any scripts which has this command.
Click to expand...
It's probably one of the periodic(8) scripts. And looking at the find command you posted it's /etc/periodic/security/100.chksetuid.
 
SirDice said:
It's probably one of the periodic(8) scripts. And looking at the find command you posted it's /etc/periodic/security/100.chksetuid.
Click to expand...

You are also right, I saw it. I will search this file 100.chksetuid. May it cause the kmem to fill?
 
Actually there are possible problems with ZFS and the periodic(8) scripts running massive find(1) runs. There are few threads here on the forums about it but don't have the time to search for them now.
 
I am experiencing similar behaviour on a machine running on ZFS that was recently upgraded from release 9.1 to release 9.2. Prior to the upgrade all was running rock solid. Now it keeps panicing almost every night at around 03:02 while running periodic daily scripts.

So far I have tried limiting the ZFS ARC size, which seems to have improved the stability to some degree, but I still get nightly panics occasionally.
 
mickey said:
I am experiencing similar behaviour on a machine running on ZFS that was recently upgraded from release 9.1 to release 9.2. Prior to the upgrade all was running rock solid. Now it keeps panicing almost every night at around 03:02 while running periodic daily scripts.

So far I have tried limiting the ZFS ARC size, which seems to have improved the stability to some degree, but I still get nightly panics occasionally.
Click to expand...

What is your kmem_size and RAM size?

In your /boot/loader.conf:
Code: 
vm.kmem_size="?"
vm.kmem_size_max="?"
Increase these properties and wait in front of the computer screen about 3 A.M. What did it write before panic on video display screen?
 
In addition I solved my problem with these codes;
Code: 
vm.kmem_size_max="8G"
vm.kmem_size="6G"
vfs.zfs.arc_min="1024M"
vfs.zfs.arc_max="1500M"
vfs.zfs.prefetch_disable=0

Before I changed kmem_size and kmem_size_max was 1 GB. Then I changed and solved my problem.
 
ShyRain said:
In addition I solved my problem with these codes;
Code: 
vm.kmem_size_max="8G"
vm.kmem_size="6G"
vfs.zfs.arc_min="1024M"
vfs.zfs.arc_max="1500M"
vfs.zfs.prefetch_disable=0

Before I changed kmem_size and kmem_size_max was 1 GB. Then I changed and solved my problem.
Click to expand...

Remove/comment out those two kmem_size* settings completely from /boot/loader.conf and let the kernel auto-tune suitable values for them, you're not supposed to change them unless you really know what you're doing.

Edit: What I'm saying applies only to AMD64 version of FreeBSD (it is clear the system in question is AMD64 from the amounts of memory quoted), on i386 the situation is different and manual tweaking of kmem_size* tunables is almost mandatory to get ZFS working properly.
 
ShyRain said:
What is your kmem_size and RAM size?

In your /boot/loader.conf:
Code: 
vm.kmem_size="?"
vm.kmem_size_max="?"
Increase these properties and wait in front of the computer screen about 3 A.M. What did it write before panic on video display screen?
Click to expand...
The machine runs FreeBSD/i386 and has 1G RAM. Up to the point where I upgraded it to 9.2, it was running smoothly with these settings:
Code: 
vm.kmem_size="512M"
vm.kmem_size_max=512M"
ARC size was automatically chosen by the kernel to be around 320M in size.

When the problems started after upgrading to 9.2, I first tried limiting the ARC size to 256M and then 192M when it still crashed. It seemed to have improved slightly, but the machine would still eventually crash at 03:02 during daily scripts, or an svn up on /usr/ports.

As limiting the ARC size did not seem to solve the problem, I am now trying out a new kernel, recompiled with:
Code: 
KVA_PAGES=512
And in my /boot/loader.conf:
Code: 
vm.kmem_size="1G"
vm.kmem_size_max="1G"
On first try, I again let the kernel auto-determine the ARC size, and it was around 640M then. Unfortunately the machine crashed again after a few hours uptime. Now I added:
Code: 
vfs.zfs.arc_max="512M"
The machine is up for 1+04:05:19 now, has survived the last daily run, and some stress I put upon it, by running some svnsync and svn up commands. Probably too early to make any assumptions about stability, but this is what it currently looks like:
Code: 
last pid: 17745;  load averages:  0.07,  0.02,  0.05    up 1+04:07:56  18:17:22
56 processes:  5 running, 51 sleeping
CPU:  0.4% user,  0.0% nice,  0.4% system,  0.0% interrupt, 99.2% idle
Mem: 93M Active, 65M Inact, 698M Wired, 135M Free
ARC: 512M Total, 286M MFU, 181M MRU, 32K Anon, 7707K Header, 37M Other
Swap: 4096M Total, 4096M Free
The only panic I could get a hold off on the console so far happened during svn up. From what I remember:
Code: 
fatal trap 12
supervisor read page not present
Not sure whether this has any influence, but FreeBSD on this machine was built from source using clang.
 
You must log in or register to reply here.
Back
Top