FreeBSD 13.1 extremally slow

C

cmiu147

Hi guys,

I have a home/nas server running on FreeBSD for over 1year... Here is the config:

Mainboard: Intel S1200BTL (intel C202 chipset)
CPU: Intel Xeon E3-1200v2
RAM: 16GB
hard drives: 4x Seagate Barracuda ST2000DM008
filesystem: ZFS
Raid: ZFS raid10

Even when I open ssh to connect to this it takes couple of second until I get my login prompt.
Now I just run iocage list to list my iocages and the result came after 30seconds. On iocage I have a cage with apache24+mariadb+php and most of the time the response time is awful.

The server is connected to the network using an Intel 1Gbps card.

Another problem is that the CPU is 99% in idle, is not even used... something is not working as expected, but i don't know how to investigate. Can you help me with some advice?


The hard drives are SMR drives. I haven't known about SMR drives when i build this machine. Could this be the answer for very slow performances?


Thank you!
 
Also DNS related, where you have more than one name server. If you your primary name server is not responding, clients will run very slowly as every DNS lookup will be delayed by waiting for a timeout on the primary server before trying the secondary server. The usual fix is to use a caching name server on each client.
 
and the delay to iocage list...that takes 30s in the first place and then takes less than 5seconds ?

why my all apps are fantastic slow compared to a linux box using the same config?
 
They're not slow. You just have something incorrectly configured, and my guess is name resolving. Yes, even on a local network this is important. Doesn't matter how you do it, DNS or distributing hosts(5) files, but the fact remains this is important to configure correctly.
 
Well...definitely something is not working as it should...
I checked the host file... everything fine there...

127.0.0.1 localhost localhost.localdomain
192.168.0.119 nfs nfs.mydomain.com
192.168.0.10 dev2 dev2.mydomain.com

where mydomain is the name of the real domain name.

what else should i look for? this was working fine couple of months ago... i haven't changed the config...

how can i identify if is a problem with my SATA controller?
 
Code: 
search mydomainname.com
#nameserver 192.168.0.1
nameserver 8.8.8.8
Code: 
# $FreeBSD$
#
group: compat
group_compat: nis
hosts: files dns
netgroup: compat
networks: files
passwd: compat
passwd_compat: nis
shells: files
services: compat
services_compat: nis
protocols: files
rpc: files

also i deleted a 200mb db and took 9minutes.

Code: 
 Query OK, 112 rows affected (9 min 12.069 sec)

i guess the hardware is not used at his full potential... but i don't understand why...
 
what's the usage of your zpool? - at least 20% free space? Is your server swapping - did you limit zfs's ARC? Whats the chipset of your network card and your ifconfig configuration (offloading? tso etc)? Do your ssh connection initiate faster if you have "UseDNS no" in your /etc/ssh/sshd_config?
 
Code: 
zpool list
NAME     SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
backup  1.81T  1.28T   540G        -         -    10%    70%  1.00x    ONLINE  -
zroot   3.62T  2.85T   795G        -         -    22%    78%  1.00x    ONLINE  -

Code: 
ZFS Subsystem Report                            Tue Oct 11 17:46:34 2022
------------------------------------------------------------------------

System Information:

        Kernel Version:                         1301000 (osreldate)
        Hardware Platform:                      amd64
        Processor Architecture:                 amd64

        ZFS Storage pool Version:               5000
        ZFS Filesystem Version:                 5

FreeBSD 13.1-RELEASE releng/13.1-n250148-fc952ac2212 GENERIC 5:46PM  up 81 days,  4:26, 2 users, load averages: 0.32, 0.28, 0.25

------------------------------------------------------------------------

System Memory:

        3.06%   477.29  MiB Active,     3.78%   590.14  MiB Inact
        89.34%  13.62   GiB Wired,      0.00%   0       Bytes Cache
        3.03%   472.43  MiB Free,       0.80%   124.27  MiB Gap

        Real Installed:                         16.00   GiB
        Real Available:                 98.02%  15.68   GiB
        Real Managed:                   97.24%  15.25   GiB

        Logical Total:                          16.00   GiB
        Logical Used:                   93.51%  14.96   GiB
        Logical Free:                   6.49%   1.04    GiB

Kernel Memory:                                  519.78  MiB
        Data:                           91.74%  476.86  MiB
        Text:                           8.26%   42.92   MiB

Kernel Memory Map:                              15.25   GiB
        Size:                           88.59%  13.51   GiB
        Free:                           11.41%  1.74    GiB

------------------------------------------------------------------------

ARC Summary: (HEALTHY)
        Memory Throttle Count:                  0

ARC Misc:
        Deleted:                                27.43   m
        Mutex Misses:                           6.15    k
        Evict Skips:                            6.83    k

ARC Size:                               76.05%  11.17   GiB
        Target Size: (Adaptive)         76.12%  11.18   GiB
        Min Size (Hard Limit):          3.34%   501.87  MiB
        Max Size (High Water):          29:1    14.68   GiB
        Compressed Data Size:                   9.61    GiB
        Decompressed Data Size:                 28.06   GiB
        Compression Factor:                     2.92

ARC Size Breakdown:
        Recently Used Cache Size:       46.75%  5.23    GiB
        Frequently Used Cache Size:     53.25%  5.95    GiB

ARC Hash Breakdown:
        Elements Max:                           980.84  k
        Elements Current:               82.90%  813.12  k
        Collisions:                             30.13   m
        Chain Max:                              8
        Chains:                                 122.26  k

------------------------------------------------------------------------

ARC Efficiency:                                 14.07   b
        Cache Hit Ratio:                99.73%  14.04   b
        Cache Miss Ratio:               0.27%   37.34   m
        Actual Hit Ratio:               99.28%  13.97   b

        Data Demand Efficiency:         97.43%  488.58  m
        Data Prefetch Efficiency:       10.14%  11.23   m

        CACHE HITS BY CACHE LIST:
          Anonymously Used:             0.35%   48.57   m
          Most Recently Used:           3.45%   484.13  m
          Most Frequently Used:         96.10%  13.49   b
          Most Recently Used Ghost:     0.05%   6.37    m
          Most Frequently Used Ghost:   0.06%   8.50    m

        CACHE HITS BY DATA TYPE:
          Demand Data:                  3.39%   476.02  m
          Prefetch Data:                0.01%   1.14    m
          Demand Metadata:              80.68%  11.32   b
          Prefetch Metadata:            15.92%  2.23    b

        CACHE MISSES BY DATA TYPE:
          Demand Data:                  33.64%  12.56   m
          Prefetch Data:                27.01%  10.09   m
          Demand Metadata:              22.49%  8.40    m
          Prefetch Metadata:            16.85%  6.29    m

------------------------------------------------------------------------

L2ARC is disabled

------------------------------------------------------------------------

Basically i haven't touched ARC parameters or zfs parameters... i only created the array, installed the OS and then configure the needed packages (samba, apache, etc).
 
When it comes to ssh annoyance make sure you have:
Code: 
grep -i usedns /etc/ssh/sshd_config
UseDNS no
Also note it does make a difference if fqdn is first in hosts or not. Fix those entries to 192.168.0.10 dev2.mydomain.com dev2 in /etc/hosts.

On Linux strace is your friend to quickly diagnose what are you waiting for. I'm sad to say this is no longer supported on FreeBSD. dtrace is a nice tool, more powerful too, but it has steep learning curve and sometimes it's harder to use for simple diagnostics.

To quickly check if DNS is culprit hash out all lines in resolv.conf so you don't have any DNS set. Make sure jails don't have that too. Then try that iocage command again. I'm assuming you can survive without DNS few minutes.
 
_martin said:
On Linux strace is your friend to quickly diagnose what are you waiting for. I'm sad to say this is no longer supported on FreeBSD. dtrace is a nice tool, more powerful too, but it has steep learning curve and sometimes it's harder to use for simple diagnostics.
Click to expand...
truss(1)
 
rootbert said:
whats the output of "zpool status", "top -b -o res", "iostat -c 5", "systat -vmstat"?
Click to expand...

Code: 
zpool status

  pool: backup
 state: ONLINE
  scan: scrub repaired 0B in 04:40:21 with 0 errors on Wed Jun 15 14:31:04 2022
config:
    NAME        STATE     READ WRITE CKSUM
    backup      ONLINE       0     0     0
      ada4      ONLINE       0     0     0

errors: No known data errors

  pool: zroot
 state: ONLINE
  scan: scrub repaired 0B in 03:07:52 with 0 errors on Tue Jul 12 16:48:40 2022
config:

    NAME        STATE     READ WRITE CKSUM
    zroot       ONLINE       0     0     0
      mirror-0  ONLINE       0     0     0
        ada0p3  ONLINE       0     0     0
        ada1p3  ONLINE       0     0     0
      mirror-1  ONLINE       0     0     0
        ada2p3  ONLINE       0     0     0
        ada3p3  ONLINE       0     0     0

errors: No known data errors

Code: 
top -b -o res
last pid:  2625;  load averages:  0.46,  0.33,  0.27; battery: -1%  up 81+06:25:52    19:45:58
53 processes:  1 running, 52 sleeping
CPU:  0.5% user,  0.0% nice,  0.3% system,  0.0% interrupt, 99.1% idle
Mem: 113M Active, 958M Inact, 19M Laundry, 14G Wired, 517M Free
ARC: 11G Total, 7453M MFU, 3146M MRU, 130K Anon, 187M Header, 658M Other
     9834M Compressed, 28G Uncompressed, 2.89:1 Ratio
Swap: 16G Total, 549M Used, 15G Free, 3% Inuse

  PID USERNAME    THR PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
 1645 mysql         8  20    0  1645M   452M select   2   0:31   0.00% mariadbd
97737 root          6  20    0   210M   126M select   2  20:59   3.42% smbd
93531 www           1  52    0   258M   122M accept   1   4:34   0.00% php-fpm
93533 www           1  52    0   274M   119M accept   1   4:33   0.00% php-fpm
93532 www           1  20    0   420M   110M accept   1   5:54   0.00% php-fpm
93545 www           1  20    0   237M   103M accept   1   4:36   0.00% php-fpm
97689 root          1  20    0   197M    37M select   2   0:02   0.00% smbd
  819 cristi        1  20    0    21M  9576K select   2   0:00   0.00% sshd
 2483 cristi        1  20    0    21M  9572K select   1   0:00   0.00% sshd
 2479 root          1  22    0    21M  9556K select   0   0:00   0.00% sshd
  815 root          1  20    0    21M  9548K select   1   0:00   0.00% sshd
18536 mysql        30  20    0  1934M  7360K select   3   1:56   0.00% mysqld
  824 root          1  20    0    16M  4752K ttyin    3   0:00   0.00% csh
 2488 root          1  20    0    16M  4588K pause    2   0:00   0.00% csh
 2625 root          1  20    0    14M  3356K CPU0     0   0:00   0.00% top
 2484 cristi        1  27    0    13M  3212K wait     0   0:00   0.00% sh
  820 cristi        1  22    0    13M  3116K wait     3   0:00   0.00% sh
 1429 mysql         1  52    0    13M  3040K wait     0   0:00   0.00% sh

Code: 
iostat -c 5
       tty            ada0             ada1             ada2             cpu
 tin  tout KB/t  tps  MB/s  KB/t  tps  MB/s  KB/t  tps  MB/s  us ni sy in id
   2     8 26.0    6   0.2  26.0    6   0.2  25.0    7   0.2   1  0  0  0 99
   0   228 18.6   62   1.1   8.9   54   0.5   7.4   38   0.3   0  0  0  0 100
   0    79  4.0    4   0.0   4.0    4   0.0  12.0   10   0.1   0  0  0  0 100
   0    79  0.0    1   0.0   4.0    2   0.0   0.0    0   0.0   0  0  0  0 100
   0    81 10.4    5   0.1   9.3    6   0.1   7.4    7   0.1   0  0  0  0 100


Code: 
    3 users    Load  0.16  0.27  0.24                  Oct 11 19:47:35
   Mem usage:  97%Phy 88%Kmem                           VN PAGER   SWAP PAGER
Mem:      REAL           VIRTUAL                        in   out     in   out
       Tot   Share     Tot    Share     Free   count
Act  1060M  74664K   5858M     182M     517M   pages
All  1066M  80744K   6005M     298M                       ioflt  Interrupts
Proc:                                                  16 cow     160 total
  r   p   d    s   w   Csw  Trp  Sys  Int  Sof  Flt       zfod        uart0 4
              89   4   567   52  641   64    9   41       ozfod     2 ehci1 20
                                                         %ozfod     2 ehci0+ 22
 0.2%Sys   0.0%Intr  0.0%User  0.0%Nice 99.8%Idle         daefr    25 cpu0:timer
|    |    |    |    |    |    |    |    |    |    |    12 prcfr    21 cpu1:timer
                                                       25 totfr    25 cpu2:timer
                                           dtbuf          react    25 cpu3:timer
Namei     Name-cache   Dir-cache    345026 maxvn          pdwak    18 em0:irq0
   Calls    hits   %    hits   %    345024 numvn       46 pdpgs    42 ahci0 37
    1373    1367 100                303793 frevn          intrn
                                                      14G wire
Disks  ada0  ada1  ada2  ada3  ada4 pass0 pass1      114M act
KB/t  12.37 11.42 12.74 12.74  0.00  0.00  0.00      958M inact
tps       9    11    11    11     0     0     0       19M laund
MB/s   0.10  0.12  0.13  0.13  0.00  0.00  0.00      517M free
%busy    30    28    53    48     0     0     0         0 buf
 
ok, I guess ARC is eating up all your memory and processes are being swapped. I recommend putting "vfs.zfs.arc_max=4294967296" into your /boot/loader.conf and reboot. This will limit your ARC to 4GB. Should you experience too much free memory afterwards you can increase that value (and reboot) to increase your ZFS cache.
 
rootbert said:
I guess ARC is eating up all your memory and processes are being swapped.
Click to expand...
There's barely any swap being used. Swap usage in and of itself isn't bad.

But I see a mariadbd and mysqld running. Those two plus ARC like to use a lot of memory. ARC and MySQL/MariaDB can and will fight over memory. So tune your databases to not use any more more memory than it actually needs and limit your ARC. And remember to keep some memory left over for the rest of your applications. With this much stuff on it I would probably opt for a memory upgrade too, 16GB isn't much.
 
  • Like
Reactions: mer
Sadly, SMR drives are effectively incompatible with ZFS. Yes, you can create a ZFS file system on them but as soon as the file system needs to write on a block that has already been written on, ZFS slows down due to the overhead in all the extra work it has to do because of the shingled recording. ZFS should only be used with Conventional Magnetic Recording (CMR) hard drives.
 
vmb said:
Sadly, SMR drives are effectively incompatible with ZFS. Yes, you can create a ZFS file system on them but as soon as the file system needs to write on a block that has already been written on, ZFS slows down due to the overhead in all the extra work it has to do because of the shingled recording. ZFS should only be used with Conventional Magnetic Recording (CMR) hard drives.
Click to expand...
Or some type of solid state storage device. ZFS is insanely fast on my NVMe boot device. I say use NVMe and if that's not pawssible, use SATA-SSDs. Come to think of it, a zraid should be on SATA-SSD.
 
You must log in or register to reply here.
Back
Top