Unexplained FreeBSD Hang

M

mickwilli

I have a FreeBSD server as a virtual machine on XCP ng running FreeBSD version 14.1-RELEASE-p6.

This is a newly installed VM and is basically only running BIND DNS server and the Bacula Director service. However it has effectively hung on me a couple of times. The first time BIND stopped responding to DNS queries for domains it did not have cached, the Bacula director had failed to start backup jobs, and the console was entirely unresponsive. The second time all services appeared to be operating normally, however I was unable to get a shell via SSH (it appeared to authenticate), and while I could login to Webmin, it didn't load the dashboard.

I can't seem to identify any good reason for this behaviour. I had a look through the messages log, and asside from some Bacula errors, due to a missing script, there was nothing out of the ordinary. From the hypervisor, it doesn't appear that more than 50% of the RAM was being used, and there was virtually no CPU usage.

Does anyone have any ideas of what might be happening here or how I could troubleshoot further?
 
I've had a quick look at the current state of these, which are as follows:

Code: 
# iostat
       tty             ada0              cd0            pass0             cpu
 tin  tout KB/t   tps  MB/s KB/t   tps  MB/s KB/t   tps  MB/s  us ni sy in id
   0    19 32.7     4  0.14  0.0     0  0.00  0.0     0  0.00   2  0  0 56 42

Code: 
# systat -> :vmstat
systat: : ambiguous request

It appears that second command isn't quite right.

Can you tell me what I'm looking at here and what would be considered normal and abnormal?
 
mickwilli said:
It appears that second command isn't quite right.
Click to expand...
Execute systat, when it opens, enter :vmstat, or just enter systat -vmstat to get there.

Code: 
     -display          The - flag expects display to be one of: icmp, icmp6,
                       ifstat, iostat, ip, ip6, netstat, pigs, sctp, swap,
                       tcp, vmstat, or zarc, These displays can also be
                       requested interactively (without the “-”) and are
                       described in full detail below.
Code: 
     Certain characters cause immediate action by systat.  These are

     ^L          Refresh the screen.

     ^G          Print the name of the current ``display'' being shown in the
                 lower window and the refresh interval.

     :           Move the cursor to the command line and interpret the input
                 line typed as a command.  While entering a command the
                 current character erase, word erase, and line kill characters
                 may be used.
Code: 
     vmstat      Take over the entire display and show a (rather crowded)
                 compendium of statistics related to virtual memory usage,
                 process scheduling, device interrupts, system name
                 translation caching, disk I/O etc.
 
According the tps output your VM is idle. Also check the hypervisor if there's high load on the storage from some other VM.
 
Thanks for the info.

Yes, I expect this VM to be virtually idle all the time - it's really not doing much. I have not specifically checked anything in the hypervisor as there are several VMs that are much much busier than this one that aren't having any issues at all (using the same storage, CPU and memory), so I doubt it's a hypervisor related issue. I should note that this is (currently) the only FreeBSD VM on this host.

The VM has been running ok for around a week now, after I gave it a bit more CPU (and maybe a little more RAM, I don't recall). Current stats from the systat command (following the instructions above):

Code: 
    1 users    Load  1.83  1.90  1.75                  Dec 26 14:38:03
   Mem usage:  96%Phy  5%Kmem                           VN PAGER   SWAP PAGER
Mem:      REAL           VIRTUAL                        in   out     in   out
       Tot   Share     Tot    Share     Free   count
Act  2094M  98544K    516G      98M     232M   pages
All  2094M  98544K    516G      98M                       ioflt  Interrupts
Proc:                                                 774 cow    1372 total
  r   p   d    s   w   Csw  Trp  Sys  Int  Sof  Flt  1493 zfod        atkbd0 1
              93       14K  11K  96K  687       37K       ozfod       uart0 4
                                                         %ozfod     1 ata1 15
 0.0%Sys  53.4%Intr  4.1%User  0.0%Nice 42.5%Idle         daefr       uhci0 23
|    |    |    |    |    |    |    |    |    |    |  2117 prcfr   294 cpu0:xen
+++++++++++++++++++++++++++>>                        5254 totfr   146 cpu1:xen
                                        23 dtbuf          react   245 cpu2:xen
Namei     Name-cache   Dir-cache    180200 maxvn          pdwak       cpu0:r
   Calls    hits   %    hits   %    180198 numvn       18 pdpgs     5 cpu0:itlb
    2814    2805 100                 73836 frevn          intrn   233 cpu0:b
                                                     908M wire        cpu1:r
Disks  ada0   cd0 pass0                               48M act       6 cpu1:itlb
KB/t   0.50  0.00  0.00                             5071M inact   109 cpu1:b
tps       0     0     0                                 0 laund       cpu2:r
MB/s   0.00  0.00  0.00                              232M free      8 cpu2:itlb
%busy     0     0     0                              541M buf     172 cpu2:b
                                                                   51 xen_et0:c0
                                                                   26 xen_et0:c1
                                                                   64 xen_et0:c2
                                                                      xenstore0
                                                                      xbd0 2121
                                                                    3 xn0 2122
                                                                    3 xn0 2123
                                                                    5 xn0 2124
                                                                    1 xn0 2125

And for completeness, here's the iostat output from about the same time:
Code: 
iostat
       tty             ada0              cd0            pass0             cpu
 tin  tout KB/t   tps  MB/s KB/t   tps  MB/s KB/t   tps  MB/s  us ni sy in id
   0     0 28.2     2  0.06  0.0     0  0.00  0.0     0  0.00   2  0  0 56 42

Any further thoughts?
 
too many cpu interrupts

FreshPorts -- sysutils/xe-guest-utilities: FreeBSD VM tools for Citrix XenServer and XCP

FreeBSD VM tools for Citrix XenServer and XCP
www.freshports.org www.freshports.org


sysctl kern.hz


spork

bhyve Thread 'FreeBSD 8.4 bhyve guest using ~25% CPU when idle'

In the middle of retiring some old stuff and need a holding ground, and found some issues with jailing something this old (an 8.4 jail is certainly possible, but I've hit some software that just does not behave in a jail)...

Logging into the 8.4 guest, top shows 0% CPU usage, and other than sshd, no services are running.

But on the 13.3 host, I have this:

Code: 
last pid: 50502;  load averages:  1.35,  0.72,  0.45                                                                  up 0+23:24:41  20:16:27
365 processes: 1 running, 364 sleeping
CPU:  0.6% user,  0.0% nice,  1.2% system,  0.0%...
 
Thanks for the extra info.

I have the xe-guest-utilities installed and they are working. I assume that is why you included that link?

I did a bit of looking into the kern.hz paramater and following the instructions from the handbook for FreeBSD as a Guest in VirtualBox etc, I set the kern.hz in the /boot/loader.conf file to 100. A quick reboot and I can confirm it appears to have applied the new setting:
Code: 
sysctl kern.hz
kern.hz: 100

However, the interrupts still seem very high, arguably worse (although this VM is now doing a little more work than it previously was).

Code: 
    1 users    Load  0.17  0.22  0.13                  Dec 27 12:44:46
   Mem usage:  48%Phy  1%Kmem                           VN PAGER   SWAP PAGER
Mem:      REAL           VIRTUAL                        in   out     in   out
       Tot   Share     Tot    Share     Free   count
Act  1448M    112M    521G     217M    3071M   pages
All  1461M    125M    521G     230M                       ioflt  Interrupts
Proc:                                                 294 cow     762 total
  r   p   d    s   w   Csw  Trp  Sys  Int  Sof  Flt   611 zfod        atkbd0 1
             182       739  969   2K  382    1  950       ozfod       uart0 4
                                                         %ozfod     1 ata1 15
 0.0%Sys  71.5%Intr  0.3%User  0.0%Nice 28.2%Idle         daefr       uhci0 23
|    |    |    |    |    |    |    |    |    |    |   866 prcfr   146 cpu0:xen
++++++++++++++++++++++++++++++++++++                 1359 totfr   127 cpu1:xen
                                        31 dtbuf          react   107 cpu2:xen
Namei     Name-cache   Dir-cache    180200 maxvn          pdwak       cpu0:r
   Calls    hits   %    hits   %      2372 numvn      499 pdpgs     7 cpu0:itlb
    1684    1684 100                   630 frevn          intrn    29 cpu0:b
                                                     650M wire        cpu1:r
Disks  ada0   cd0 pass0                             1183M act       7 cpu1:itlb
KB/t   0.50  0.00  0.00                             1042M inact    38 cpu1:b
tps       0     0     0                                 0 laund       cpu2:r
MB/s   0.00  0.00  0.00                             3071M free      4 cpu2:itlb
%busy     0     0     0                              500M buf      46 cpu2:b
                                                                  107 xen_et0:c0
                                                                   77 xen_et0:c1
                                                                   54 xen_et0:c2
                                                                    1 xenstore0
                                                                      xbd0 2121
                                                                    4 xn0 2122
                                                                    3 xn0 2123
                                                                    3 xn0 2124
                                                                    1 xn0 2125

Code: 
iostat
       tty             ada0              cd0            pass0             cpu
 tin  tout KB/t   tps  MB/s KB/t   tps  MB/s KB/t   tps  MB/s  us ni sy in id
   0   311 35.4   164  5.68  0.0     0  0.00  0.0     0  0.00   5  0  0 69 26
 
I can offer you to write little shell script to dump information like memory, CPU, iostat in log files and run it with cron every minute.
Maybe it could help to analyze problem
 
You must log in or register to reply here.
Back
Top