Solved strange behaviour while copy files

wolffnx

wolffnx

Hi, I dont know how describe this but here I go

the motherboard is a asus pbz77-m , I have 2 6GB sata ports and 4 3GB sata ports
the main disk is a solid ssd , connected to the 6GB port
this disk give me :
smartctl -a /dev/ada0
Code: 
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
everything is good

then, I have 2 hard disks of 2TB in mirror mode
smartctl gives me
Code: 
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 3.0 Gb/s)
good..in the two drives

the problem is when I copy files from the pool in the mirror disks
starts copy the file, then dont copy nothing for a while and then start again
the output of
iostat -w1

Code: 
       tty            ada0             ada1             ada2             cpu
 tin  tout KB/t  tps  MB/s  KB/t  tps  MB/s  KB/t  tps  MB/s  us ni sy in id
   5   334  120  289  33.9   764   23  17.5   753   23  17.2   0  0  1  0 99
   1   545  111  555  60.3   982  128 122.7   985  127 122.0   0  0  6  0 93
   0   191  128  945 118.0   968  131 123.5   984  131 125.5   0  0  4  0 96
   0   193  128 1511 188.8  1002  131 128.7  1015  127 126.3   0  0  4  0 96
   0   192  128  722  90.0  1009  132 129.9  1006  130 127.5   0  0  6  0 94
   0   192  128  638  79.6   993  131 127.0  1023  127 126.7   0  0  4  0 96
   0   191  111  657  71.1  1009  130 128.3  1024  128 128.2   0  0  8  0 92
   0   193  128  609  76.0   987  131 126.2  1022  126 125.6   0  0  3  0 97
       tty            ada0             ada1             ada2             cpu
 tin  tout KB/t  tps  MB/s  KB/t  tps  MB/s  KB/t  tps  MB/s  us ni sy in id
   0   191  128  522  65.3   988   34  32.7  1024   36  35.9   0  0  1  0 99
   0   533  128  658  82.1   0.0    0   0.0   0.0    0   0.0   0  0  1  0 99
   0   192  128  577  72.0   0.0    0   0.0   0.0    0   0.0   0  0  1  0 99
   0   190  128  700  87.5   0.0    0   0.0   0.0    0   0.0   0  0  0  0 100
   0   194  127  554  68.9   0.0    0   0.0   0.0    0   0.0   0  0  0  0 100
   0   328  128  544  68.0   0.0    0   0.0   0.0    0   0.0   0  0  1  0 99
   0   198  128  505  63.0   0.0    0   0.0   0.0    0   0.0   0  0  0  0 100
   0   197  128  476  59.5   0.0    0   0.0   0.0    0   0.0   0  0  0  0 100
       tty            ada0             ada1             ada2             cpu
 tin  tout KB/t  tps  MB/s  KB/t  tps  MB/s  KB/t  tps  MB/s  us ni sy in id
   1   189  128  531  66.2   0.0    0   0.0   0.0    0   0.0   0  0  1  0 99
   0   561  128  470  58.8   0.0    0   0.0   0.0    0   0.0   0  0  0  0 99
   0   187  128  517  64.4   0.0    0   0.0   0.0    0   0.0   0  0  0  0 100
^C   8   545  128  367  45.9   0.0    0   0.0   0.0    0   0.0   0  0  0  0 99

(ada0 is the main system)

only have in
/boot/loader.conf
the following lines
Code: 
kern.cam.pmp.default_timeout=30
kern.hz="300"
vfs.zfs.arc.max="3G"
fusefs_load="YES"
#cryptodev_load="YES"
#ichsmb_load="YES"
#smbus_load="YES"
hw.usb.no_boot_wait="1"
hw.usb.no_shutdown_wait=1
kern.cam.boot_delay="500"
kern.cam.scsi_delay="500"
vfs.zfs.prefetch_disable="0"

I take out the other modules..nvidia,etc

any ideas?
 
richardtoohey2 said:
Caching somewhere?

Does the copy succeed?
Click to expand...

yes,at least copy the file/s , but take too much time
but if I want to cancel the copy process, sometimes I can , and other times dont

update:
I install FreeBSD on another ssd disk from scratch and import the pool of 2TB
and put only one disk to the other 6GB port
(only the main OS and one of the mirroed disks)
and have same behaviour
 
What is the consumption/state of your RAM during the process?

And the state of ARC, you can see the status in a simple way with a package that you have in the ports called zfs-stats.
 
freejlr said:
What is the consumption/state of your RAM during the process?

And the state of ARC, you can see the status in a simple way with a package that you have in the ports called zfs-stats.
Click to expand...
I never use zfs-stats , but during the copy give me this results (gnu-watch -n1 zfs-stats -a)

Code: 
ARC Summary: (HEALTHY)                                                                                                           
        Memory Throttle Count:                  0                                                                               
                                                                                                                                
ARC Misc:                                                                                                                       
        Deleted:                                65                                                                               
        Mutex Misses:                           0                                                                               
        Evict Skips:                            2.50    k                                                                       
                                                                                                                                
ARC Size:                               50.35%  13.55   GiB                                                                     
        Target Size: (Adaptive)         50.63%  13.62   GiB                                                                     
        Min Size (Hard Limit):          3.24%   893.00  MiB                                                                     
        Max Size (High Water):          30:1    26.91   GiB                                                                     
        Compressed Data Size:                   12.90   GiB                                                                     
        Decompressed Data Size:                 12.99   GiB                                                                     
        Compression Factor:                     1.01                                                                             
                                                                                                                                
ARC Size Breakdown:                                                                                                             
        Recently Used Cache Size:       96.22%  13.11   GiB                                                                     
        Frequently Used Cache Size:     3.78%   527.06  MiB                                                                     
                                                                                                                                
ARC Hash Breakdown:                                                                                                             
        Elements Max:                           109.03  k                                                                       
        Elements Current:               100.00% 109.03  k                                                                       
        Collisions:                             1.46    k                                                                       
        Chain Max:                              2                                                                               
        Chains:                                 1.40    k

this time X freeze for a couple of seconds(6 or 8) , even wont gime the option to switch to the console, and seconds later,everyting back to normaly, is strange
 
Smells a bit like caching to me - there's a big build-up of data to be written, the drives report they are getting the data but then at certain points they have to go flat out writing the data they've cached and they can't accept any more data until they have written their buffered/cached data.

So you see lots of write progress, then it seems to stall (because the drives are saying they are busy busy busy) then the drives catch up and report themselves ready for more data.

But that's just a theory!
 
wolffnx said:
(ada0 is the main system)
Click to expand...
Yes, and that ada0 is moving some 60-80 MB per second continuousely. That is quite a lot for a system disk. I'd figure out what it is doing, as the first step.

Run gstat -p. That is not so much suited for cut&paste, but it will tell you
- if the load is read or write
- how much saturated that disk is.
Because, if you see a disk that is some 90-100% busy, then the system may likely stall on that activity.

wolffnx said:
the problem is when I copy files from the pool in the mirror disks
Click to expand...

BTW: to where?
Your iostat printout shows there is 2 x 128 MB per second transferred from ada1+ada2. To where? Where is that data going?
 
PMc said:
Yes, and that ada0 is moving some 60-80 MB per second continuousely. That is quite a lot for a system disk. I'd figure out what it is doing, as the first step.

Run gstat -p. That is not so much suited for cut&paste, but it will tell you
- if the load is read or write
- how much saturated that disk is.
Because, if you see a disk that is some 90-100% busy, then the system may likely stall on that activity.



BTW: to where?
Your iostat printout shows there is 2 x 128 MB per second transferred from ada1+ada2. To where? Where is that data going?
Click to expand...

from ada1/ada2 to ada0
ada0 is the ssd main OS and ada1/ada2 are the mirror disk of 2TB
 
Thanks PMc
the bottleneck is in ada0 , when I start copy the file the disk usage is 100% and ada1 and ada2 are waiting
more easy if you see it:


 
The problem is the motherboard, never happend this in ANY motherboard, update the firmware too, but no results...I'l wait for a answer here and if not..get back to my old mobo
 
Your analysis of ad0 writing slower than ada1+2 reading seems correct by posted iostat traffic. If ada0 is idle at the same time, then there may be an issue to look into further. Submitted videos don't view for me as they are marked as private. Writes are normally delayed to a ZFS pool by `sysctl vfs.zfs.txg.timeout` seconds, though less once the cache gets filled too much and reads will stop until there is enough room to store them. That timeout was reduced from 30 seconds to 5 seconds years ago; I increase it back on mine as I noticed no difference in system usability and under heavy throughput I found ZFS waited less than 5 seconds before writing.

When transferring a lot of data, many SSDs have a small cache that takes data in very quickly to higher quality+speed storage and then writes it out to the slower+cheaper memory so it can appear to have a burst of write speed look good but get worse under sustained write activity. This could explain a faster initial write speed going to a slower write speed. Knowing the SSDs and firmware could help others know if performance is as expected or not.

If motherboard BIOS and drives firmware are updated with results still lower than desired then you may still gain performance by trimming the ssd if autotrim is off and the drive is not new or had plenty of its total amount of free space previously written to. The motherboard being likely over 10 years old makes working with the manufacturer to get any new BIOS fixes beyond what was released in 2016 unlikely.

It wouldn't speed up the drive but could speed up data throughput if data is compressible by enabling compression or turn it up to a higher setting until CPU/RAM becomes a bottleneck of performance.
 
Mirror176 said:
Your analysis of ad0 writing slower than ada1+2 reading seems correct by posted iostat traffic. If ada0 is idle at the same time, then there may be an issue to look into further. Submitted videos don't view for me as they are marked as private. Writes are normally delayed to a ZFS pool by `sysctl vfs.zfs.txg.timeout` seconds, though less once the cache gets filled too much and reads will stop until there is enough room to store them. That timeout was reduced from 30 seconds to 5 seconds years ago; I increase it back on mine as I noticed no difference in system usability and under heavy throughput I found ZFS waited less than 5 seconds before writing.

When transferring a lot of data, many SSDs have a small cache that takes data in very quickly to higher quality+speed storage and then writes it out to the slower+cheaper memory so it can appear to have a burst of write speed look good but get worse under sustained write activity. This could explain a faster initial write speed going to a slower write speed. Knowing the SSDs and firmware could help others know if performance is as expected or not.

If motherboard BIOS and drives firmware are updated with results still lower than desired then you may still gain performance by trimming the ssd if autotrim is off and the drive is not new or had plenty of its total amount of free space previously written to. The motherboard being likely over 10 years old makes working with the manufacturer to get any new BIOS fixes beyond what was released in 2016 unlikely.

It wouldn't speed up the drive but could speed up data throughput if data is compressible by enabling compression or turn it up to a higher setting until CPU/RAM becomes a bottleneck of performance.
Click to expand...
thanks for you point of view, I decide to back to my other motherboard, and asrock h61m-dgs
the "problem" is gone, you right, the motherboard is too old, and in that times maybe they dont consider the presence of ssd disks
 
You must log in or register to reply here.
Back
Top