Solved Very slow performance on Thinkpad T440p (ZFS?)

dnb

dnb

On my Thinkpad T440p laptop performance is very poor. Even xterm opens with a slight lag. And this despite the fact that the xmonad window manager is used. I must say that in general this is a pretty peppy laptop with top configuration for 2015 (i7 haswell + 16gb ram + ssd samsung pro), if you just put some Linux distribution like Gentoo.

How I set up the system:
  • used legacy bios (default fot this model, uefi available);
  • booted from live-cd to console;
  • created zpool (zfs mirror);
  • installed zfsboot bootloader without any disk partitioning;
  • unpacked base and kernel from the archive;
  • built packages on a separate server (binary host) using poudriere and put them here on T440p.
1 - Some hardware information:
Code: 
Version: Intel(R) Core(TM) i7-4710MQ CPU @ 2.50GHz
        Voltage: 0.8 V                                      
        External Clock: 100 MHz    
        Max Speed: 2500 MHz                                
        Current Speed: 2500 MHz      
        Status: Populated, Enabled          
        Upgrade: Socket rPGA988B                          
        L1 Cache Handle: 0x0002        
        L2 Cache Handle: 0x0003                
        L3 Cache Handle: 0x0004                  
        Serial Number: None                                
        Asset Tag: None                                    
        Part Number: None                                  
        Core Count: 4                                                                                                      
        Core Enabled: 4                                    
        Thread Count: 8                                    
        Characteristics:                                    
                64-bit capable

RAM: 16Gb
HDD: SSD Samsung Pro 128Gb x2 (ZFS mirror) + Transcend 128Gb (l2arc)

2 - ZFS. I am using legacy bios and zfsboot. There is no disk partitioning, ZFS occupies the entire disk.

Code: 
root@t440p:~ # zpool status
  pool: t440p
 state: ONLINE
  scan: resilvered 18.4G in 00:04:08 with 0 errors on Wed Oct 12 11:21:31 2022
config:

        NAME                             STATE     READ WRITE CKSUM
        t440p                            ONLINE       0     0     0
          mirror-0                       ONLINE       0     0     0
            diskid/DISK-S24ZNXAH111111N  ONLINE       0     0     0
            diskid/DISK-S24ZNXAH222222K  ONLINE       0     0     0
        cache
          diskid/DISK-G553111111         ONLINE       0     0     0

errors: No known data errors

/etc/sysctl.conf
Code: 
kern.coredump=0
kern.randompid=1
net.inet.ip.forwarding=1
vfs.zfs.min_auto_ashift=12 # use sectors more than 4KiB

I tried to remove the cache (l2arc) from the pool, and I also removed the second disk from the mirror, leaving it with one disk, no cache and no redundancy. It had virtually no effect on performance. The problem is floating, sometimes the browser can be used, but sometimes even updating the page takes time. It feels like there is not enough RAM. What is your opinion about the causes of this problem?
 
16GB of ram is more than enough for zfs without any tuning. ( I have 8GB)
Maybe there is a problem in the freebsd-kernel for your CPU ?
Can you show the output of
Code: 
top -n 10 -o res
 
  • Thanks
Reactions: dnb
Alain De Vos said:
16GB of ram is more than enough for zfs without any tuning. ( I have 8GB)
Maybe there is a problem in the freebsd-kernel for your CPU ?
Can you show the output of
Code: 
top -n 10 -o res
Click to expand...
Code: 
root@t440p:~ # top -n 10 -o res
last pid: 96898;  load averages:  0.24,  0.19,  0.17; battery: 99%  up 0+00:50:33    21:14:14
27 processes:  1 running, 26 sleeping
CPU:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle
Mem: 33M Active, 33M Inact, 410M Wired, 15G Free
ARC: 137M Total, 41M MFU, 48M MRU, 400K Anon, 46M Header, 2218K Other
     60M Compressed, 102M Uncompressed, 1.70:1 Ratio

  PID USERNAME    THR PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
24513 root          1  20    0    21M  9288K select   0   0:00   0.00% sshd
13086 root          1  20    0    21M  8448K select   6   0:00   0.00% sshd
25819 root          1  20    0    19M  7764K ttyin    3   0:00   0.00% csh
50822 root          1  20    0    16M  7520K pause    6   0:00   0.00% csh
24783 root          1  52    0    16M  7520K pause    3   0:00   0.00% csh
92844 root          1  46    0    63M  7420K kqread   7   0:00   0.00% cupsd
14327 root          1  20    0    21M  7364K select   6   0:00   0.00% wpa_supplicant
96766 root          1  20    0    21M  6672K select   3   0:00   0.00% ntpd
71289 root         14 -44   r8    18M  6192K cuse-s   5   0:00   0.00% webcamd
25767 root          1  20    0    15M  5232K select   3   0:00   0.00% tmux

P.S.: but this is ssh session, xorg even not started.

And now with xorg + firefox:
Code: 
root@t440p:~ # top -n 10 -o res
last pid: 15044;  load averages:  0.66,  0.30,  0.21; battery: 99%  up 0+00:59:20    21:23:01
41 processes:  1 running, 40 sleeping
CPU:  0.2% user,  0.0% nice,  0.0% system,  0.0% interrupt, 99.8% idle
Mem: 800M Active, 544M Inact, 17M Laundry, 1257M Wired, 13G Free
ARC: 645M Total, 433M MFU, 151M MRU, 760K Anon, 47M Header, 8568K Other
     543M Compressed, 622M Uncompressed, 1.14:1 Ratio

  PID USERNAME    THR PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
11632 m            82  26    0  3210M   674M select   5   0:24  33.89% firefox
11947 m            36  22    0    11G   556M select   3   0:18  50.88% firefox
11874 m            28  20    0  2588M   200M select   0   0:00   0.00% firefox
14184 m             4  20    0   494M   188M select   1   0:00   0.00% firefox
14138 m            19  20    0  2514M   188M select   0   0:00   0.00% firefox
12167 m            20  20    0  2518M   186M select   0   0:00   0.00% firefox
12125 m            20  20    0  2518M   186M select   3   0:00   0.00% firefox
11852 m            21  20    0  2522M   171M select   3   0:00   0.00% firefox
11820 m             7  20    0   344M   150M select   1   0:00   0.00% firefox
 5506 m             3  20    0   213M   106M select   7   0:01   0.00% Xorg
 
You have 0% CPU & 15G Free Memory. In "theory" everything should be fast ...
Can you try openbox or fluxbox & viewing a youtube video with firefox-esr or chromium ?
 
  • Thanks
Reactions: dnb
Which FreeBSD version are you using?

Just to be sure, did you set vfs.zfs.min_auto_ashift=12 before you created your mirror pool?
 
  • Thanks
Reactions: dnb
Erichans said:
Which FreeBSD version are you using?
Click to expand...
Code: 
root@t440p:~ # cat /etc/os-release
NAME=FreeBSD
VERSION="13.1-RELEASE-p2"
VERSION_ID="13.1"
ID=freebsd
ANSI_COLOR="0;31"
PRETTY_NAME="FreeBSD 13.1-RELEASE-p2"
CPE_NAME="cpe:/o:freebsd:freebsd:13.1"
HOME_URL="https://FreeBSD.org/"
BUG_REPORT_URL="https://bugs.FreeBSD.org/"
Erichans said:
Just to be sure, did you set vfs.zfs.min_auto_ashift=12 before you created your mirror pool?
Click to expand...
Perhaps my mistake is somewhere here. I just created the pool with zpool create -m none t440p /dev/diskid/......

I didn't do sysctl vfs.zfs.min_auto_ashift=12 on this system. I simply transferred sysctl.conf from system with a similar configuration. If this is my mistake, then I have to check all my machines, because somewhere I could do sysctl vfs.zfs.min_auto_ashift=12 before creating the pool, but on other machines I did not. Is there any way to determine what the vfs.zfs.min_auto_ashift settings were when the pool was created?
 
Alain De Vos said:
Code: 
sysctl -a | grep ashift
could be interesting.
Click to expand...
I have already removed the vfs.zfs.min_auto_shift=12 setting on this system and rebooted.
Code: 
root@t440p:~ # sysctl -a | grep ashift
vfs.zfs.max_auto_ashift: 16
vfs.zfs.min_auto_ashift: 9
vfs.zfs.vdev.max_auto_ashift: 16
vfs.zfs.vdev.min_auto_ashift: 9
vfs.zfs.vdev.file.physical_ashift: 9
vfs.zfs.vdev.file.logical_ashift: 9
 
When in the future you should make a new zpool set,
Code: 
vfs.zfs.min_auto_ashift=12

Yet the ashift cannot explain bursts of slowness with 0%CPU usage & 15GB Free Memory. It must be something else.
For graphics you might need "drm-510-kmod"
 
  • Thanks
Reactions: dnb
Alain De Vos said:
When in the future you should make a new zpool set,
Code: 
vfs.zfs.min_auto_ashift=12
Click to expand...
Excuse me, I would like to ask again. Do I understand correctly that when I create a new pool on this machine, the algorithm of my actions will be as follows:
  1. Determine the sector size (all disks in the pool are the same for me).
  2. Set settings in /etc/sysctl.conf.
For example,
Determine the sector size:
Code: 
root@t440p:~ # diskinfo -c /dev/diskid/DISK-S24ZNXAH111111K
/dev/diskid/DISK-S24ZNXAH111111K
        512             # sectorsize
        128035676160    # mediasize in bytes (119G)
        250069680       # mediasize in sectors
        4096            # stripesize
        0               # stripeoffset
        248085          # Cylinders according to firmware.
        16              # Heads according to firmware.
        63              # Sectors according to firmware.
        Samsung SSD 850 PRO 128GB       # Disk descr.
        S24ZNXAH111111Ks0       # Disk ident.
        ahcich5         # Attachment
        id1,enc@n3061686369656d30/type@0/slot@6/elmdesc@Slot_05 # Physical path
        Yes             # TRIM/UNMAP support
        0               # Rotation rate in RPM

I/O command overhead:
        time to read 10MB block      0.023168 sec       =    0.001 msec/sector
        time to read 20480 sectors   2.015167 sec       =    0.098 msec/sector
        calculated command overhead                     =    0.097 msec/sector

Sector size - 512, so I shouldn't change anything in vfs.zfs.min_auto_ashift at all. Right?
On all my machines the sector size is 512, so I didn't have to specify vfs.zfs.min_auto_ashift anywhere. It was my mistake, I copied the setting from some manual without fully understanding it. Moreover, now I have to recreate all pools. Correctly?
 
Erichans said:
ashift is an individual VDEV property, set at creation time. Try zdb(8): zdb -C <poolname> and look at the children entries.
Click to expand...
Yes, I have already deleted vfs.zfs.min_auto_ashift from /etc/sysctl.conf and rebooted, but I still have 12:
Code: 
root@t440p:~ # zdb -C t440p | grep ashift
                ashift: 12

Is it better to re-create the zpool?
 
T-Daemon said:
No.

zpoolprops(7)
Code: 
The following properties can be set at creation time and import time, and
     later changed with the zpool set command:

     ashift=ashift
Click to expand...
I set ashift with zpool set ashift=9 t440p, but zdb still show 12:
Code: 
root@t440p:~ # zdb -C t440p | grep ashift
                ashift: 12
                ashift: 0
                ashift: 0
 
is the lag perceptible in console/text mode ?
you may turn off atime but probably wont change much regardless to the lag
try
/usr/bin/time find /usr/src/contrib -type f|xargs cat>/dev/null
 
  • Thanks
Reactions: dnb
dnb said:
I set ashift with zpool set ashift=9 t440p, but zdb still show 12:
Code: 
root@t440p:~ # zdb -C t440p | grep ashift
                ashift: 12
                ashift: 0
                ashift: 0
Click to expand...
Once set at creation, ashift is an immutable VDEV property, this is what your quoted zdb -C t440p shows. Have a look at Tuning recordsize in OpenZFS - March 30, 2022 by Klara systems:
Ashift is a vdev-wide setting, must be set when creating each vdev, and is immutable once set—it cannot be manipulated afterward, or on a per-dataset basis. If a vdev makes it into a pool with a bad ashift value—for example, a Samsung SSD which lies to ZFS and claims to use 512-byte sectors, and an admin who doesn’t manually specify -o ashift=13 when creating a vdev with it—the only recourse is destroying the pool and rebuilding it from scratch.
Click to expand...
zpoolprops(7) (my emphasis):
Rich (BB code): 
ashift=ashift
	     Pool sector size exponent,	to the power of	2 (internally referred
	     to	as ashift).  Values from 9 to 16, inclusive, are valid;	also,
	     the value 0 (the default) means to	auto-detect using the kernel's
	     block layer and a ZFS internal exception list.  I/O operations
	     will be aligned to	the specified size boundaries.	Additionally,
	     the minimum (disk)	write size will	be set to the specified	size,
	     so	this represents	a space	vs. performance	trade-off.  For	opti-
	     mal performance, the pool sector size should be greater than or
	     equal to the sector size of the underlying	disks.	The typical
	     case for setting this property is when performance	is important
	     and the underlying	disks use 4KiB sectors but report 512B sectors
	     to	the OS (for compatibility reasons); in that case, set
	     ashift=12 (which is 1<<12 = 4096).	 When set, this	property is
	     used as the default hint value in subsequent vdev operations
	     (add, attach and replace).	 Changing this value will not modify
	     any existing vdev,	not even on disk replacement; however it can
	     be	used, for instance, to replace a dying 512B sectors disk with
	     a newer 4KiB sectors device: this will probably result in bad
	     performance but at	the same time could prevent loss of data.
SSDs are in the habit of lying about their sectorsize. Some info/discussion about that with a list of SSDs and their sector size: Feature request: change ashift default value to 12. Your Samsung PRO 850 isn't mentioned in that list unfortunately. Based on what what I know, it would be unlikely that you should use an ashift of 9 for a modern SSD; a value of 12 or 13 is common. Set the appropriate value for vfs.zfs.min_auto_ashift* before a VDEV creation or use for example -o ashift=13 explicitely at the command line at the time of VDEV creation.

Given that your ashift is currently set at 12**, my guess is that that the source of your "speed" problem is most likely not your ashift value, be the optimum value 12 or 13. I don't have any further particular suggestions for resolving your speed problem.

___
* Be aware that there are "older" spinning rust platters (especially the smaller ones) that may actually have 512 byte sectors; even though modern versions of spinning rust platters have a) actually 4096 byte sectors and report that accurately or b) have 4096 byte sectors and report them (in-accurately) as having 512 bytes sectors.
** ZFS 101—Understanding ZFS storage and performance: "If ashift is set too low, an astronomical read/write amplification penalty is incurred—writing a 512 byte "sectors" to a 4KiB real sector means having to write the first "sector", then read the 4KiB sector, modify it with the second 512 byte "sector", write it back out to a *new* 4KiB sector, and so forth, for every single write."
 
Last edited:
  • Thanks
Reactions: dnb
You must log in or register to reply here.
Back
Top