Solved System hangs when uploading large numbers of files via NFS4 or Samba (root cause: realtek driver)

striker2150

New Member

Reaction score: 6
Messages: 14

Hi,

I am running a small FreeBSD DIY NAS. When I upload large amounts of data (for example, photo galleries with several hundreds of pictures including RAW files), the copy process often hangs. Sometimes, it continues after a while and sometimes I have to reboot the FreeBSD box. Copy processes are started on Windows10 clients using Samba or Debian 10 using NFSv4.

Code:
# sysctl hw.model hw.machine hw.ncpu hw.physmem
hw.model: Intel(R) Pentium(R) CPU  J3710  @ 1.60GHz
hw.machine: amd64
hw.ncpu: 4
hw.physmem: 8001310720

Data is stored on a zpool consisting of three identical 4TB WD Red harddisks configured as raidz1.
ZFS dataset compression and deduplication are turned off.

Memory shouldn't be an issue.
However, I am not sure if the CPU could be a limiting factor and hinder FreeBSD to write data quickly on disk.

Can you suggest any parameters for tuning the system?
Goal would be to have stable copy processes even if the write rate slows down a bit.

Thanks and best regards
 

ralphbsz

Son of Beastie

Reaction score: 2,030
Messages: 2,997

Before we tune, let's fix the breakage. You say "hang". Does it come back out of hang? Yes, you say sometimes it continues after a while. When it is in the "hang" state, could it be that it is just doing work, and has slowed down? Do you have disk lights that are blinking? Is there lots of network activity? What happens to other processes that are already running? Can you log in (but it is very slow)?

Suggestion: When you bring the system up, connect a console to it, and keep "top" running on it. If you have more than one console, also run "vmstat 1" and "iostat 1" in a few extra consoles. Then, when it hangs, see what they say.
 
OP
striker2150

striker2150

New Member

Reaction score: 6
Messages: 14

First, sorry for the late answer.

My impression is that it often just slows down and recovers.
But there were also cases without any process after waiting for an hour or more.

I will try running top, vmstat 1, and iostat 1.
 

fcorbelli

Member

Reaction score: 31
Messages: 93

I would also check the zfs properties, in particular the existence or not of compression, and also the use of the swapfile (it's easy, from top).

It should be remembered that RAIDZ1 loads the machine's CPU, personally I never recommend it (in favor of the normal mirror) for such a small installation.

It is possible (just a hypothesis) that the ARC cache is filled gradually, and then written gradually to disk and this is the slowdown phase.
I would also try making a copy directly from a different device (ideally a test SSD) to observe the same behavior/not
 
OP
striker2150

striker2150

New Member

Reaction score: 6
Messages: 14

Top results during copy operation via the network:
Code:
system: up to ~26%
interrupt: up to ~18%
idle: ~50-97%
swap: 32M used 8159M free
So, does not look as if the CPU would be under heavy load.

Iostat outputs show that there are short writes on the disks (ada1, ada2, and ada3) of approx 12,5 MB/s and then the numbers returning to zero again.
So short burst and then nothing for a while.
That is not happening if I start a local copy operation, e.g., from the system disk to the RAIDZ1.

There was no hang during this test run, just a slow copy operation via the network (~6.9 MB/s).
Both computers are connected via gigabit ethernet with almost no other activity, so I would expect a copy speed of maybe 40-110 MB/s (just in theory up to 125MB/s for gigabit ethernet).

I made an interesting observation. When replacing my gigabit ethernet switch (1000 Mbit) with an fast ethernet switch (100 Mbit), the copy process went up to 11.0 MB/s.
That is very close to the theoretical maximum of 12.5 MB/s and much more than with the faster gigabit switch.
What do you think, might there be a issue with the network hardware?
Maybe replacing the cables would be an idea (both are just 2 meter cable length and therefore usually should not be an issue).

I am not quite sure about what parameters to observe on vmstat.
Maybe you can give me a tip on that.
 

T-Daemon

Daemon

Reaction score: 696
Messages: 1,466

striker2150, you didn't mention the FreeBSD version you are using on the NAS.

There is a recent started conversation on freebsd-current@ about unresponsive NFSv4/3 connection for several minutes to the clients, or total failure to recover in a case with a Ubuntu client. Your problem might be related, maybe the same as described there:


See also the review mentioned in the mails:
 
OP
striker2150

striker2150

New Member

Reaction score: 6
Messages: 14

@T-Deamon
When I wrote the first post, the system was running on FreeBSD 12.2.
However, I upgraded to FreeBSD 13 early after release.
The issue was not solved by the OS upgrade.
I did not perform a zpool upgrade.

This is what my iostat output looks like if the copy process hangs (ada1-3 are the raidz disks):
Code:
       tty            ada1             ada2             ada3             cpu
 tin  tout KB/t  tps  MB/s  KB/t  tps  MB/s  KB/t  tps  MB/s  us ni sy in id
   0   157  0.0    0   0.0   0.0    0   0.0   0.0    0   0.0   1  0  1  1 97
   0  1647  0.0    0   0.0   0.0    0   0.0   0.0    0   0.0   1  0  1  0 98
   0   160 64.0    2   0.1   0.0    0   0.0  64.0    2   0.1   2  0  2  0 96
   0  1806  0.0    0   0.0   0.0    0   0.0   0.0    0   0.0   2  0  1  0 97
   0   149  0.0    0   0.0   0.0    0   0.0   0.0    0   0.0   1  0  1  0 98
   0  2252  0.0    0   0.0   0.0    0   0.0   0.0    0   0.0   1  0  1  0 97
   0   161 64.0    2   0.1   0.0    0   0.0  64.0    2   0.1   1  0  1  0 98
   0  2078  0.0    0   0.0   0.0    0   0.0   0.0    0   0.0   0  0  1  0 98
   0   158  0.0    0   0.0   0.0    0   0.0   0.0    0   0.0   1  0  1  0 98
   0  1891  0.0    0   0.0   0.0    0   0.0   0.0    0   0.0   1  0  1  1 97
   0   160 64.0    1   0.1  64.0    1   0.1  64.0    2   0.1   1  0  1  0 98
   0  1563  0.0    0   0.0   0.0    0   0.0   0.0    0   0.0   1  0  1  0 97
   0   156  0.0    0   0.0   0.0    0   0.0   0.0    0   0.0   1  0  1  0 98
   0  1469  0.0    0   0.0   0.0    0   0.0   0.0    0   0.0   1  0  1  0 98
   0   158  0.0    0   0.0   0.0    0   0.0   0.0    0   0.0   1  0  1  0 98
   0  1707  0.0    0   0.0  64.0    2   0.1  64.0    2   0.1   0  0  2  0 98
   0   157  0.0    0   0.0   0.0    0   0.0   0.0    0   0.0   1  0  1  0 97
   0  2102  0.0    0   0.0   0.0    0   0.0   0.0    0   0.0   1  0  1  0 98
   0   157  0.0    0   0.0  64.0    2   0.1  64.0    2   0.1   1  0  1  0 97
   0  2057  0.0    0   0.0   0.0    0   0.0   0.0    0   0.0   1  0  1  0 98
   0   154 64.0    1   0.1  64.0    1   0.1  64.0    2   0.1   1  0  1  0 98
   0  2214  0.0    0   0.0   0.0    0   0.0   0.0    0   0.0   1  0  1  0 98
   0   311  0.0    0   0.0   0.0    0   0.0   0.0    0   0.0   1  0  1  0 98
   0  1717 64.0    2   0.1   0.0    0   0.0  64.0    2   0.1   0  0  1  0 98
   0   157  0.0    0   0.0   0.0    0   0.0   0.0    0   0.0   1  0  1  0 97
   0  1660 64.0    2   0.1   0.0    0   0.0  64.0    2   0.1   2  0  1  0 96
   0   157  0.0    0   0.0   0.0    0   0.0   0.0    0   0.0   1  0  1  0 98
   0  1386  0.0    0   0.0   0.0    0   0.0   0.0    0   0.0   1  0  1  0 97
   0   157  0.0    0   0.0   0.0    0   0.0   0.0    0   0.0   1  0  1  0 98
   0  1894  0.0    0   0.0  64.0    2   0.1  64.0    2   0.1   2  0  1  0 97
   0   157  6.2  114   0.7   6.2  115   0.7   6.3  115   0.7   1  0  2  1 97
   0  1682  0.0    0   0.0   0.0    0   0.0   0.0    0   0.0   1  0  1  0 98

I will see if iperf brings new insights.
Furthermore, I try another gigabit switch and cables.

Thanks so far for all the tips.
 
OP
striker2150

striker2150

New Member

Reaction score: 6
Messages: 14

That is how copy operations from a local UFS disk (ada0) to the RAIDZ array look like:

Code:
       tty            ada1             ada2             ada3             cpu
 tin  tout KB/t  tps  MB/s  KB/t  tps  MB/s  KB/t  tps  MB/s  us ni sy in id
   0   158  0.0    0   0.0  64.0    4   0.2  64.0    4   0.2   2  0 19  3 77
   0   314  307  100  30.0   294  111  31.8   307  115  34.5   1  0 27  2 70
   0   158  147  322  46.4   201  226  44.3   221  194  41.9   1  0 26  3 71
   0   159  3.2    5   0.0   3.2    5   0.0   2.7    6   0.0   2  0 12  3 84
   0   159  187  357  65.2   432  145  61.2   310  213  64.5   1  0 34  3 62
   0   158 49.6  228  11.0   110  142  15.2  80.3  153  12.0   2  0 18  5 76
   0   155  153   63   9.4   302   29   8.4   327   18   5.7   1  0 20  2 78
   0    79  123  568  68.2   210  337  69.0   270  274  72.1   1  0 31  5 63
   0   159  2.7    6   0.0   2.7    6   0.0   2.7    6   0.0   1  0 14  4 81
   0   158  617   72  43.4   669   61  39.8   675   64  42.2   2  0 24  4 70
   0   158  124  271  32.8   203  184  36.6   198  177  34.2   1  0 25  5 70
   0   158 64.0    1   0.1  64.0    3   0.2  64.0    4   0.2   1  0 13  3 83
   0   158  171  457  76.3   230  339  76.2   319  245  76.4   2  0 39  5 54
   0   159  9.8   37   0.4   8.9   28   0.2  12.3   28   0.3   2  0 17  3 78
   0   158  479   92  43.0   591   69  39.8   598   68  39.7   0  0 26  5 69
   0   158  109  308  32.9   242  154  36.3   164  227  36.4   1  0 28  2 69
   0   158  132  591  76.2   287  272  76.1   269  290  76.1   1  0 36  3 60
   1   169  2.7    6   0.0  25.5   11   0.3  25.1   11   0.3   2  0 17  2 79

I did this copy operation to examine if the error is really related to the RAIDZ1 pool or has something to do with the network.
So, a local copy operation seems to be fine.

Next, I configured a Samba share to share a folder located on the UFS disk.
Downloading data from the UFS drive to windows 10 was without any issues (approx 50-70 MB/s).
Uploading the data from Win10 to the UFS drive via Samba caused the same issue.
The copy process started with approx 10-20 MB/s and hung after coping 4% of the data.
I had to kill the Explorer application via the task manager as it was frozen.

I used approx 2.8 GB of test data consisting of 90 CR2 files and 90 JPG files for the test.
That is the typical data mix you will get after a short photo shooting.

So, maybe I was wrong with blaming the RAIDZ array and it is more a network issue.
 

fcorbelli

Member

Reaction score: 31
Messages: 93

The CPU has a low level of single-threaded performance, this may simply be the reason.

If you can try to install some prepackaged "NAS" software, for example truenas / freenas.
There you are sure that you have no configuration problems or miscellaneous conflicts, and you will see the performance reasonably achievable.

You can also try napp-it, which is based on Solaris (much more basic interface, you need to get used to zfs, but it's essentially very similar) if you really want to get rid of any doubts.

Another test to do is, trivially, DO NOT use RAIDZ, but single disks or, at most, mirrored.
The CPU performance issue is, for me, the first to clarify.
 

rootbert

Well-Known Member

Reaction score: 139
Messages: 382

whats is the chipset of your network interface? maybe you can test some ifconfig parameters like (-)tso, (-)lro, (-)rxcsum etc
 
OP
striker2150

striker2150

New Member

Reaction score: 6
Messages: 14

rootbert your question solved my problem. Thanks a lot.

The ASRock J3710-ITX mainboard is equipped with a Realtek RTL8111GR network chip.
I found that the FreeBSD Realtek driver seems to cause problems.

https://forums.freebsd.org/threads/realtek-rtl8111g-unstable.71825/

Problem was solved by installing the realtek-re-kmod package.

pkg install realtek-re-kmod
vi /boot/loader.conf


Put the following stuff into loader.conf:

Code:
if_re_load="YES"
if_re_name="/boot/modules/if_re.ko"
hw.re.max_rx_mbuf_sz="2048"

After rebooting the system, I was able to upload data with a constant copy rate of 52 MB/s.
I have to check if the hw.re.max_rx_mbuf_sz setting is really necessary.
Maybe I get a better performance without.

Anyway, thanks to everybody for your help!
 
OP
striker2150

striker2150

New Member

Reaction score: 6
Messages: 14

Informational Update:

I removed the hw.re.max_rx_mbuf_sz setting from loader.conf but did not experience any change.
Copy process was still stable at approx 52 MB/s.
 
Top