Slow HDD performance

Shura · Jun 28, 2013

Hello

After update to 9.1-RELEASE amd64 I have been faced with slow HDD performance.

I don't know how to measure the performance, but when I run VLC, Conky displays HDD read speed ~ 2.75[ ]Mb. It is very few, but the HDD LED is bright and I hear the HDD crunch. If I ran three copies of VLC, disc reading increases up to ~4.15[ ]Mb/s and it starts to drop frames and print in the console:

Code:

[0x801fa2d58] main input error: ES_OUT_RESET_PCR called
[0x801fa2d58] main input error: ES_OUT_SET_(GROUP_)PCR  is called too late (pts_delay increased to 1000 ms)
[0x801fa2d58] main input error: ES_OUT_RESET_PCR called

top

Code:

51043 shura         5  22    0   213M 35364K uwait   0   0:16  5.27% vlc
51058 shura         5  22    0   213M 34392K uwait   0   0:15  4.98% vlc
51054 shura         5  22    0   217M 35748K uwait   1   0:15  4.88% vlc

So, reading speed is very slow.

Code:

$ mount | grep ada0
/dev/ada0p2 on / (ufs, local, journaled soft-updates)

Please suggest (a) command(s) to diagnose the problem.

Shura · Jun 28, 2013

Hm, seems like it is not an HDD problem. The FTP server on this host can handle 50[ ]Mb/s. Maybe it is a VLC problem.

mav@ · Jun 28, 2013

Try to use something simple as dd if=file of=/dev/null bs=1m for testing.

wblock@ · Jun 28, 2013

diskinfo(8) has a convenient read-only test: diskinfo -tv ada0.

programmerdams · Oct 21, 2013

HDD busy at low MB/s

Were you able to find a fix? I've been getting a 100% busy ( systat -vmstat) at 3-40 MB/s. It's quite variable within that range. The HDD is a 6 GB/s Seagate 7200 1 TB. It used be give me much more than 100 MB/s with FreeBSD 9.0. It also hasn't improved after I upgraded to 9.2. I can also see my HDD indicator glowing during these instances. I do get normal transfer speeds with dd. SMART is clean, since it's a brand new drive (hardly about 3-4 months old).

wblock@ · Oct 21, 2013

On newer or larger drives, make certain the filesystem partitions are aligned to 4K blocks. The output of gpart show can be used:

Code:

% gpart show ada0
=>       34  500118125  ada0  GPT  (238G)
         34          6        - free -  (3.0k)
         40       1024     1  freebsd-boot  (512k)
       1064        984        - free -  (492k)
       2048    4194304     2  freebsd-ufs  (2.0G)
    4196352    4194304     3  freebsd-ufs  (2.0G)
    8390656  491727496     4  freebsd-ufs  (234G)
  500118152          7        - free -  (3.5k)

Both the 512-byte starting block and the size of each partition should be evenly divisible by 8, because there are eight 512-byte blocks in a single 4K block.

40 / 8 = 5
2048 / 8 = 256
4196352 / 8 = 524544
8390656 / 8 = 1048832

programmerdams · Oct 22, 2013

Code:

$ gpart show
=>        34  1953525101  ada0  GPT  (931G)
          34         128     1  freebsd-boot  (64k)
         162  1944059776     2  freebsd-ufs  (927G)
  1944059938     8388608     3  freebsd-swap  (4.0G)
  1952448546     1076589        - free -  (525M)

Looks like it isn't 4k aligned.

Code:

Model Family:     Seagate Barracuda 7200.12
Device Model:     ST31000524AS
Serial Number:    9VPGHR85
LU WWN Device Id: 5 000c50 04ea9f7c5
Firmware Version: JC45
User Capacity:    1,000,204,886,016 bytes [1.00 TB]
Sector Size:      512 bytes logical/physical
Rotation Rate:    7200 rpm
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ATA8-ACS T13/1699-D revision 4
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Tue Oct 22 05:52:41 2013 IST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

Why does smartctl report it as a 512k drive?

programmerdams · Oct 22, 2013

Is there a way to re-partition my disk without data loss?
And, could someone link me to a tutorial that could guide me through this (possibly without formatting)? I'll keep backups, either case.

wblock@ · Oct 22, 2013

That is a drive with only 512-byte blocks. It does not need to be aligned to 4K blocks. Also, Seagate firmware in the 4K drives has an auto-align method that seems pretty effective, so even on those drives it is not a huge issue.

Deleted member 13721 · Oct 22, 2013

wblock@ said:
That is a drive with only 512-byte blocks. It does not need to be aligned to 4K blocks. Also, Seagate firmware in the 4K drives has an auto-align method that seems pretty effective, so even on those drives it is not a huge issue.

Apparently some drives lie about their sectoring, claiming to have 512-byte blocks but actually using 4k. I have a drive with "advanced format", initially I didn't pay much attention to it, but during a recent refit of my system, I realized it was 4k and needed to be reconfigured properly. So I went through the process of dumping and restoring partitions and rebuilding /etc/fstab with cat. It was quite an adventure.

It must of worked because the stutter in I/O intensive tasks has completely disappeared.

wblock@ · Oct 22, 2013

At present, all 4K drives say they use 512-byte blocks. If the physical sectors are larger than the logical sectors, it will show in the output of gpart list with the sectorsize and stripesize. Although as @programmerdams showed above, it's actually a little easier to see in the smartctl output.

kpa · Oct 22, 2013

The reason why drives still report 512 byte blocks is that there is no BIOS out there that could boot from a disk with 4KB sectors, only UEFI could boot from such disks as far as I know.

Shura · Oct 22, 2013

I did not find fix.

Code:

=== START OF INFORMATION SECTION ===
Model Family:     Seagate Barracuda 7200.9
Device Model:     ST3250824AS
Serial Number:    4ND1TXNA
Firmware Version: 3.AAD
User Capacity:    250 059 350 016 bytes [250 GB]
Sector Size:      512 bytes logical/physical
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ATA/ATAPI-7 (minor revision not indicated)
Local Time is:    Tue Oct 22 10:35:06 2013 MSK
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

Code:

$ gpart show ada0
=>       34  488397101  ada0  GPT  (232G)
         34        128     1  freebsd-boot  (64k)
        162  478150528     2  freebsd-ufs  (228G)
  478150690    8388608     3  freebsd-swap  (4.0G)
  486539298    1857837        - free -  (907M)

Seems like partitions are not aligned to 4K blocks.

But reading test is ok:

Code:

$ sudo diskinfo -tv ada0
ada0
        512             # sectorsize
        250059350016    # mediasize in bytes (232G)
        488397168       # mediasize in sectors
        0               # stripesize
        0               # stripeoffset
        484521          # Cylinders according to firmware.
        16              # Heads according to firmware.
        63              # Sectors according to firmware.
        4ND1TXNA        # Disk ident.

Seek times:
        Full stroke:      250 iter in   5.119021 sec =   20.476 msec
        Half stroke:      250 iter in   3.716592 sec =   14.866 msec
        Quarter stroke:   500 iter in   6.119912 sec =   12.240 msec
        Short forward:    400 iter in   2.301989 sec =    5.755 msec
        Short backward:   400 iter in   2.126573 sec =    5.316 msec
        Seq outer:       2048 iter in   0.202805 sec =    0.099 msec
        Seq inner:       2048 iter in   0.180605 sec =    0.088 msec
Transfer rates:
        outside:       102400 kbytes in   1.544152 sec =    66315 kbytes/sec
        middle:        102400 kbytes in   1.765468 sec =    58002 kbytes/sec
        inside:        102400 kbytes in   2.963196 sec =    34557 kbytes/sec

VLC still have slow performance.

programmerdams · Oct 22, 2013

Kernel Issue?

Could it be a kernel issue? Because I never noticed any problem with 9.0. I'm pretty sure about this as I moved to FreeBSD after my old WD failed (I used to use Debian, previously.)

Why do I get good dd speeds, but poor random access performance? Some fault in kernel file system handling? And do I really need to align my drives, from that data posted?

wblock@ · Oct 23, 2013

That does seem slow, but maybe not out of line for that particular drive.

Not a problem here, but when you run a speed test, run it on the partitions, because those are what can be misaligned. The whole disk will always be aligned. So diskinfo -tv ada0p2 rather than just ada0.

programmerdams · Oct 23, 2013

Code:

$ diskinfo -tv /dev/ada0p2
/dev/ada0p2
        512             # sectorsize
        995358605312    # mediasize in bytes (927G)
        1944059776      # mediasize in sectors
        0               # stripesize
        82944           # stripeoffset
        1928630         # Cylinders according to firmware.
        16              # Heads according to firmware.
        63              # Sectors according to firmware.
        9VPGHR85        # Disk ident.

Seek times:
        Full stroke:      250 iter in   8.495470 sec =   33.982 msec
        Half stroke:      250 iter in   6.801952 sec =   27.208 msec
        Quarter stroke:   500 iter in   9.786443 sec =   19.573 msec
        Short forward:    400 iter in   4.697413 sec =   11.744 msec
        Short backward:   400 iter in   3.157056 sec =    7.893 msec
        Seq outer:       2048 iter in   0.584182 sec =    0.285 msec
        Seq inner:       2048 iter in   0.142617 sec =    0.070 msec
Transfer rates:
        outside:       102400 kbytes in   5.308956 sec =    19288 kbytes/sec
        middle:        102400 kbytes in   4.371047 sec =    23427 kbytes/sec
        inside:        102400 kbytes in   5.483029 sec =    18676 kbytes/sec

Code:

$ diskinfo -tv /dev/ada0p3
/dev/ada0p3
        512             # sectorsize
        4294967296      # mediasize in bytes (4.0G)
        8388608         # mediasize in sectors
        0               # stripesize
        3221242880      # stripeoffset
        8322            # Cylinders according to firmware.
        16              # Heads according to firmware.
        63              # Sectors according to firmware.
        9VPGHR85        # Disk ident.

Seek times:
        Full stroke:      250 iter in   2.730686 sec =   10.923 msec
        Half stroke:      250 iter in   4.352162 sec =   17.409 msec
        Quarter stroke:   500 iter in   4.851111 sec =    9.702 msec
        Short forward:    400 iter in   1.996861 sec =    4.992 msec
        Short backward:   400 iter in   5.224189 sec =   13.060 msec
        Seq outer:       2048 iter in   0.135682 sec =    0.066 msec
        Seq inner:       2048 iter in   0.128252 sec =    0.063 msec
Transfer rates:
        outside:       102400 kbytes in   4.173061 sec =    24538 kbytes/sec
        middle:        102400 kbytes in   8.204391 sec =    12481 kbytes/sec
        inside:        102400 kbytes in   4.200728 sec =    24377 kbytes/sec

Are both mal-aligned? And can they be aligned without expecting much data loss?

Code:

$ gpart show
=>        34  1953525101  ada0  GPT  (931G)
          34         128     1  freebsd-boot  (64k)
         162  1944059776     2  freebsd-ufs  (927G)
  1944059938     8388608     3  freebsd-swap  (4.0G)
  1952448546     1076589        - free -  (525M)

Deleted member 13721 · Oct 23, 2013

programmerdams said:

Code:

$ diskinfo -tv /dev/ada0p2
/dev/ada0p2
        512             # sectorsize
        995358605312    # mediasize in bytes (927G)
        1944059776      # mediasize in sectors
        0               # stripesize
        82944           # stripeoffset
        1928630         # Cylinders according to firmware.
        16              # Heads according to firmware.
        63              # Sectors according to firmware.
        9VPGHR85        # Disk ident.

Seek times:
        Full stroke:      250 iter in   8.495470 sec =   33.982 msec
        Half stroke:      250 iter in   6.801952 sec =   27.208 msec
        Quarter stroke:   500 iter in   9.786443 sec =   19.573 msec
        Short forward:    400 iter in   4.697413 sec =   11.744 msec
        Short backward:   400 iter in   3.157056 sec =    7.893 msec
        Seq outer:       2048 iter in   0.584182 sec =    0.285 msec
        Seq inner:       2048 iter in   0.142617 sec =    0.070 msec
Transfer rates:
        outside:       102400 kbytes in   5.308956 sec =    19288 kbytes/sec
        middle:        102400 kbytes in   4.371047 sec =    23427 kbytes/sec
        inside:        102400 kbytes in   5.483029 sec =    18676 kbytes/sec

Code:

$ diskinfo -tv /dev/ada0p3
/dev/ada0p3
        512             # sectorsize
        4294967296      # mediasize in bytes (4.0G)
        8388608         # mediasize in sectors
        0               # stripesize
        3221242880      # stripeoffset
        8322            # Cylinders according to firmware.
        16              # Heads according to firmware.
        63              # Sectors according to firmware.
        9VPGHR85        # Disk ident.

Seek times:
        Full stroke:      250 iter in   2.730686 sec =   10.923 msec
        Half stroke:      250 iter in   4.352162 sec =   17.409 msec
        Quarter stroke:   500 iter in   4.851111 sec =    9.702 msec
        Short forward:    400 iter in   1.996861 sec =    4.992 msec
        Short backward:   400 iter in   5.224189 sec =   13.060 msec
        Seq outer:       2048 iter in   0.135682 sec =    0.066 msec
        Seq inner:       2048 iter in   0.128252 sec =    0.063 msec
Transfer rates:
        outside:       102400 kbytes in   4.173061 sec =    24538 kbytes/sec
        middle:        102400 kbytes in   8.204391 sec =    12481 kbytes/sec
        inside:        102400 kbytes in   4.200728 sec =    24377 kbytes/sec

Are both mal-aligned? And can they be aligned without expecting much data loss?

Code:

$ gpart show
=>        34  1953525101  ada0  GPT  (931G)
          34         128     1  freebsd-boot  (64k)
         162  1944059776     2  freebsd-ufs  (927G)
  1944059938     8388608     3  freebsd-swap  (4.0G)
  1952448546     1076589        - free -  (525M)

Did you use bsdinstall for this? I'm not sure if it auto aligns to 4k. If not you'll have to do it by hand. Your disk performance values seem slow though.

wblock@ · Oct 23, 2013

Repeat: alignment can only be a problem if the drive actually has 4K physical blocks. The output shown says it does not have 4K blocks, and I have not found any information from Seagate that claims otherwise.

Also, Seagate's on-board firmware is supposed to avoid alignment problems. If it is 4K, and it has that firmware, anyway.

First, make a backup. Second, show the full output of smartctl -a /dev/ada0. After that, I'd run the SMART long test, because that drive is suspiciously slow on some tests and ridiculously slow on others.

Deleted member 13721 · Oct 23, 2013

wblock@ said:
Repeat: alignment can only be a problem if the drive actually has 4K physical blocks. The output shown says it does not have 4K blocks, and I have not found any information from Seagate that claims otherwise.

Also, Seagate's on-board firmware is supposed to avoid alignment problems. If it is 4K, and it has that firmware, anyway.

First, make a backup. Second, show the full output of smartctl -a /dev/ada0. After that, I'd run the SMART long test, because that drive is suspiciously slow on some tests and ridiculously slow on others.

I can't find any evidence that it's a 4k drive either. x(

Also the auto-alignment didn't seem to work for me, I had performance problems until I redid my partitions and explicitly specified them to use 4k sectors. Now that we're discussing this topic, it is possible to have proper alignment with a 4k drive and a MBR scheme right? Someone in the mailing list claimed so, but I just want clarification on that.

J65nko · Oct 23, 2013

Most FreeBSD versions, at least the ones I tried, insist on 63 sector boundary. That is why I align the MBR partitions on multiples of 63 x 8= 504 sectors.

wblock@ · Oct 23, 2013

I'll start a new thread about MBR and alignment, as it doesn't belong in this one.

Edit: Thread 42725

programmerdams · Oct 24, 2013

Here's my S.M.A.R.T. output:

Code:

smartctl 6.2 2013-07-26 r3841 [FreeBSD 9.2-RELEASE amd64] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, [url=http://www.smartmontools.org]www.smartmontools.org[/URL]

=== START OF INFORMATION SECTION ===
Model Family:     Seagate Barracuda 7200.12
Device Model:     ST31000524AS
Serial Number:    9VPGHR85
LU WWN Device Id: 5 000c50 04ea9f7c5
Firmware Version: JC45
User Capacity:    1,000,204,886,016 bytes [1.00 TB]
Sector Size:      512 bytes logical/physical
Rotation Rate:    7200 rpm
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ATA8-ACS T13/1699-D revision 4
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Thu Oct 24 06:40:05 2013 IST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82)    Offline data collection activity
                    was completed without error.
                    Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0)    The previous self-test routine completed
                    without error or no self-test has ever 
                    been run.
Total time to complete Offline 
data collection:         (  600) seconds.
Offline data collection
capabilities:              (0x7b) SMART execute Offline immediate.
                    Auto Offline data collection on/off support.
                    Suspend Offline collection upon new
                    command.
                    Offline surface scan supported.
                    Self-test supported.
                    Conveyance Self-test supported.
                    Selective Self-test supported.
SMART capabilities:            (0x0003)    Saves SMART data before entering
                    power-saving mode.
                    Supports SMART auto save timer.
Error logging capability:        (0x01)    Error logging supported.
                    General Purpose Logging supported.
Short self-test routine 
recommended polling time:      (   1) minutes.
Extended self-test routine
recommended polling time:      ( 170) minutes.
Conveyance self-test routine
recommended polling time:      (   2) minutes.
SCT capabilities:            (0x103f)    SCT Status supported.
                    SCT Error Recovery Control supported.
                    SCT Feature Control supported.
                    SCT Data Table supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   119   099   006    Pre-fail  Always       -       222746778
  3 Spin_Up_Time            0x0003   100   100   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -       347
  5 Reallocated_Sector_Ct   0x0033   100   100   036    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000f   076   060   030    Pre-fail  Always       -       49365900
  9 Power_On_Hours          0x0032   094   094   000    Old_age   Always       -       5581
 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       346
183 Runtime_Bad_Block       0x0032   100   100   000    Old_age   Always       -       0
184 End-to-End_Error        0x0032   100   100   099    Old_age   Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
188 Command_Timeout         0x0032   100   099   000    Old_age   Always       -       4
189 High_Fly_Writes         0x003a   100   100   000    Old_age   Always       -       0
190 Airflow_Temperature_Cel 0x0022   064   058   045    Old_age   Always       -       36 (Min/Max 30/36)
194 Temperature_Celsius     0x0022   036   042   000    Old_age   Always       -       36 (0 26 0 0 0)
195 Hardware_ECC_Recovered  0x001a   038   023   000    Old_age   Always       -       222746778
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       11
240 Head_Flying_Hours       0x0000   100   253   000    Old_age   Offline      -       141003776334121
241 Total_LBAs_Written      0x0000   100   253   000    Old_age   Offline      -       2983518881
242 Total_LBAs_Read         0x0000   100   253   000    Old_age   Offline      -       1089607631

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%      5478         -
# 2  Extended offline    Completed without error       00%      5290         -
# 3  Extended offline    Completed without error       00%      4434         -
# 4  Extended offline    Completed without error       00%      1705         -
# 5  Short offline       Completed without error       00%      1701         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

No errors were logged (for a drive with such low performance)
As I have mentioned previously, I do get good dd speeds.
I'd also like to mention that those extended tests were done less than 6 months back, as it's a brand new drive.

Slow HDD performance

Shura

Shura

mav@

wblock@

programmerdams

wblock@

programmerdams

programmerdams

wblock@

Deleted member 13721

Guest

wblock@

kpa

Shura

programmerdams

wblock@

programmerdams

Deleted member 13721

Guest

wblock@

Deleted member 13721

Guest

J65nko

wblock@

programmerdams