Solved zfs write speed limited to <1mb/s

R

rihad

Hi. We have two identical SSD disks:
Code: 
ada0 at ahcich4 bus 0 scbus5 target 0 lun 0
ada0: <INTEL SSDSC2BB480G6 G2010140> ACS-2 ATA SATA 3.x device
ada0: Serial Number PHWxxxxxxxxxxxxFGN
ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 512bytes)
ada0: Command Queueing enabled
ada0: 457862MB (937703088 512 byte sectors)
ada0: Previously was known as ad12
ada1 at ahcich5 bus 0 scbus6 target 0 lun 0
ada1: <INTEL SSDSC2BB480G6 G2010140> ACS-2 ATA SATA 3.x device
ada1: Serial Number PHWxxxxxxxxxxxxFGN
ada1: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 512bytes)
ada1: Command Queueing enabled
ada1: 457862MB (937703088 512 byte sectors)
ada1: Previously was known as ad14

ada0 works fine. ada1 worked fine too, up until recently. Its read speed is normal, but writes are limited to about 1mb/s. Here's the output of gstat so you get the idea:




Code: 
 L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w   %busy Name
    1   4154    651  40593    0.9   1929 128759    0.2   71.8| ada0
   10      8      0      0    0.0      8   1022   1280  102.1| ada1

Please note the low writes per second and low throughput, and how long each write op takes (1280 ms) for ada1 compared to fractions of ms for ada0.
Normally this drive hosts MySQL backups taken nightly by innobackupex, but it doesn't really matter, a simplee big file copy operation is enough to experience speed problems. Also, at the time the write operations are taking place it becomes impossible to ^C ^Z or kill -TERM any such process, even kill -KILL doesn't kill it right away. When no write operations are taking place the drive sits idle (as per gstat). What could be wrong with it? I've tried rebooting, too (FreeBSD 10.3).

zfs config is pretty simple and it's similar to our other drives & servers.

Code: 
$ zfs get all ssd1/xtradb-backups
NAME                 PROPERTY              VALUE                  SOURCE
ssd1/xtradb-backups  type                  filesystem             -
ssd1/xtradb-backups  creation              Fri Oct  6  8:26 2017  -
ssd1/xtradb-backups  used                  197G                   -
ssd1/xtradb-backups  available             228G                   -
ssd1/xtradb-backups  referenced            197G                   -
ssd1/xtradb-backups  compressratio         1.03x                  -
ssd1/xtradb-backups  mounted               yes                    -
ssd1/xtradb-backups  quota                 none                   default
ssd1/xtradb-backups  reservation           none                   default
ssd1/xtradb-backups  recordsize            128K                   local
ssd1/xtradb-backups  mountpoint            /var/backups/xtradb    local
ssd1/xtradb-backups  sharenfs              off                    default
ssd1/xtradb-backups  checksum              on                     default
ssd1/xtradb-backups  compression           lz4                    local
ssd1/xtradb-backups  atime                 off                    local
ssd1/xtradb-backups  devices               on                     default
ssd1/xtradb-backups  exec                  on                     default
ssd1/xtradb-backups  setuid                on                     default
ssd1/xtradb-backups  readonly              off                    default
ssd1/xtradb-backups  jailed                off                    default
ssd1/xtradb-backups  snapdir               hidden                 default
ssd1/xtradb-backups  aclmode               discard                default
ssd1/xtradb-backups  aclinherit            restricted             default
ssd1/xtradb-backups  canmount              on                     default
ssd1/xtradb-backups  xattr                 off                    temporary
ssd1/xtradb-backups  copies                1                      default
ssd1/xtradb-backups  version               5                      -
ssd1/xtradb-backups  utf8only              off                    -
ssd1/xtradb-backups  normalization         none                   -
ssd1/xtradb-backups  casesensitivity       sensitive              -
ssd1/xtradb-backups  vscan                 off                    default
ssd1/xtradb-backups  nbmand                off                    default
ssd1/xtradb-backups  sharesmb              off                    default
ssd1/xtradb-backups  refquota              none                   default
ssd1/xtradb-backups  refreservation        none                   default
ssd1/xtradb-backups  primarycache          all                    default
ssd1/xtradb-backups  secondarycache        all                    default
ssd1/xtradb-backups  usedbysnapshots       0                      -
ssd1/xtradb-backups  usedbydataset         197G                   -
ssd1/xtradb-backups  usedbychildren        0                      -
ssd1/xtradb-backups  usedbyrefreservation  0                      -
ssd1/xtradb-backups  logbias               latency                local
ssd1/xtradb-backups  dedup                 off                    default
ssd1/xtradb-backups  mlslabel                                     -
ssd1/xtradb-backups  sync                  standard               local
ssd1/xtradb-backups  refcompressratio      1.03x                  -
ssd1/xtradb-backups  written               197G                   -
ssd1/xtradb-backups  logicalused           203G                   -
ssd1/xtradb-backups  logicalreferenced     203G                   -
ssd1/xtradb-backups  volmode               default                default
ssd1/xtradb-backups  filesystem_limit      none                   default
ssd1/xtradb-backups  snapshot_limit        none                   default
ssd1/xtradb-backups  filesystem_count      none                   default
ssd1/xtradb-backups  snapshot_count        none                   default
ssd1/xtradb-backups  redundant_metadata    all                    default

Thanks for any tips.
 
How many drives do you have there (ad14?)? And how are they connected? Does the problem persist when you switch the places of the drives?
 
Also, running ls or du on any directory of that drive blocks for a long time, nothing kills it or stops it.
Pressing ^T shows that it blocks in [zfs]

Code: 
$ du -sh /var/backups/xtradb/Jul08_2018/Jul08_2018-full/
^C^Z
load: 13.18  cmd: du 48731 [zfs] 2839.08r 0.00u 0.00s 0% 2236k


its state as per ps -auxw is T+


rihad 48731 0.0 0.0 14496 2252 2 T+ 5:13PM 0:00.00 du -sh /var/backups/xtradb/Jul08_2018/Jul08_2018-full/

 
Crivens said:
How many drives do you have there (ad14?)? And how are they connected? Does the problem persist when you switch the places of the drives?
Click to expand...
We only have 2 physical drives (as per /var/run/dmesg.boot). ada0 & ada1. It's a dedicated remote server.
 
Run sysutils/smartmontools on this misbehaving Intel SSD. We had such case with a customer, where SSD's controller decided drive's lifetime is over (too much bytes written) and simply pulled the handbrake. If this is the case, there is nothing you can do, apart from replacing the drive, hopefully with a non-Intel one.
 
I've actually already run smartctl on it, and even a short self-test with no errors.

Code: 
$ sudo smartctl -a /dev/ada1
smartctl 6.6 2017-11-05 r4594 [FreeBSD 10.3-RELEASE-p29 amd64] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Intel 730 and DC S35x0/3610/3700 Series SSDs
Device Model:     INTEL SSDSC2BB480G6
Serial Number:    PHWxxxxxxxxxxxGN
LU WWN Device Id: 5 5cd2e4 04c678a1d
Firmware Version: G2010140
User Capacity:    480,103,981,056 bytes [480 GB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    Solid State Device
Form Factor:      2.5 inches
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2 T13/2015-D revision 3
SATA Version is:  SATA 2.6, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Sun Jul  8 18:33:07 2018 +04
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x02) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (    2) seconds.
Offline data collection
capabilities:                    (0x79) SMART execute Offline immediate.
                                        No Auto Offline data collection support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   1) minutes.
Extended self-test routine
recommended polling time:        (   2) minutes.
Conveyance self-test routine
recommended polling time:        (   2) minutes.
SCT capabilities:              (0x003d) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 1
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  5 Reallocated_Sector_Ct   0x0032   100   100   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   100   100   000    Old_age   Always       -       19251
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       14
170 Available_Reservd_Space 0x0033   100   100   010    Pre-fail  Always       -       0
171 Program_Fail_Count      0x0032   100   100   000    Old_age   Always       -       0
172 Erase_Fail_Count        0x0032   100   100   000    Old_age   Always       -       0
174 Unsafe_Shutdown_Count   0x0032   100   100   000    Old_age   Always       -       12
175 Power_Loss_Cap_Test     0x0033   100   100   010    Pre-fail  Always       -       5830 (115 889)
183 SATA_Downshift_Count    0x0032   100   100   000    Old_age   Always       -       0
184 End-to-End_Error        0x0033   100   100   090    Pre-fail  Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
190 Temperature_Case        0x0022   061   053   000    Old_age   Always       -       39 (Min/Max 19/50)
192 Unsafe_Shutdown_Count   0x0032   100   100   000    Old_age   Always       -       12
194 Temperature_Internal    0x0022   100   100   000    Old_age   Always       -       39
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
199 CRC_Error_Count         0x003e   100   100   000    Old_age   Always       -       0
225 Host_Writes_32MiB       0x0032   100   100   000    Old_age   Always       -       45099013
226 Workld_Media_Wear_Indic 0x0032   100   100   000    Old_age   Always       -       102400
227 Workld_Host_Reads_Perc  0x0032   100   100   000    Old_age   Always       -       32
228 Workload_Minutes        0x0032   100   100   000    Old_age   Always       -       1155076
232 Available_Reservd_Space 0x0033   100   100   010    Pre-fail  Always       -       0
233 Media_Wearout_Indicator 0x0032   001   001   000    Old_age   Always       -       0
234 Thermal_Throttle        0x0032   100   100   000    Old_age   Always       -       0/0
241 Host_Writes_32MiB       0x0032   100   100   000    Old_age   Always       -       45099013
242 Host_Reads_32MiB        0x0032   100   100   000    Old_age   Always       -       21740919
243 NAND_Writes_32MiB       0x0032   100   100   000    Old_age   Always       -       55016353

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%     18963         -
# 2  Short offline       Completed without error       00%        33         -
# 3  Short offline       Completed without error       00%        31         -
# 4  Short offline       Completed without error       00%        31         -
# 5  Short offline       Completed without error       00%        26         -
# 6  Short offline       Completed without error       00%        26         -
# 7  Short offline       Completed without error       00%         0         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
 
You may have a point. Host_Writes_32MiB is 45099013 (45 million) for this drive, as opposed to 5118272 (5 million) for ada0 (identical disks).
 
Well, how about Media_Wearout_Indicator 001? I believe that stands for remaining lifetime in %. Here is a sample from a customer's production server:
Code: 
Model Family:     Intel 730 and DC S35x0/3610/3700 Series SSDs
233 Media_Wearout_Indicator 0x0032   096   096   000    Old_age   Always       -       0
233 Media_Wearout_Indicator 0x0032   093   093   000    Old_age   Always       -       0
233 Media_Wearout_Indicator 0x0032   092   092   000    Old_age   Always       -       0
233 Media_Wearout_Indicator 0x0032   095   095   000    Old_age   Always       -       0
233 Media_Wearout_Indicator 0x0032   091   091   000    Old_age   Always       -       0
233 Media_Wearout_Indicator 0x0032   094   094   000    Old_age   Always       -       0
233 Media_Wearout_Indicator 0x0032   097   097   000    Old_age   Always       -       0
233 Media_Wearout_Indicator 0x0032   098   098   000    Old_age   Always       -       0
 
Bingo.
Media_Wearout_Indicator is 001
for ada1, and 087 for ada0
So Intel decided to put a brake on it indeed.
 
You must log in or register to reply here.
Back
Top