22TB Western Digital WD221KFGX shows up as 2TB

scotia

scotia

Hi all,
I recently bought a 22TB WD WD221KFGX, and on my FreeBSD 13.-1-RELEASE system it shows up as a 2TB drive.
The drive is in an enclosure connected to a LSI controller:
Code: 
# uname -a
FreeBSD freebsd-test-01 13.1-RELEASE FreeBSD 13.1-RELEASE releng/13.1-n250148-fc952ac2212 GENERIC amd64
The enclosure is attached to mpt1:
Code: 
# dmesg | grep -E 'mpt'
mpt0: <LSILogic SAS/SATA Adapter> port 0x4000-0x40ff mem 0xfd5ec000-0xfd5effff,0xfd5f0000-0xfd5fffff irq 18 at device 0.0 on pci3
mpt0: MPI Version=1.5.0.0
mpt1: <LSILogic SAS/SATA Adapter> port 0x6000-0x60ff mem 0xfd3ec000-0xfd3effff,0xfd3f0000-0xfd3fffff irq 16 at device 0.0 on pci5
mpt1: MPI Version=1.5.20.0
The drive is detected by the CAM:
Code: 
# camcontrol devlist
<NECVMWar VMware IDE CDR00 1.00>   at scbus0 target 0 lun 0 (cd0,pass0)
<VMware Virtual disk 2.0>          at scbus2 target 0 lun 0 (da0,pass1)
<DELL SC200 1.03>                  at scbus3 target 6 lun 0 (pass2,ses0)
<SEAGATE ST4000NM0023 GE0F>        at scbus3 target 7 lun 0 (pass6,da4)
...
<ATA WDC WD221KFGX-68 0A83>        at scbus3 target 19 lun 0 (pass11,da9)
...
<TOSHIBA MG03SCA300 DG02>          at scbus3 target 27 lun 0 (pass8,da6)
It's an old adapter:
Code: 
# mptutil -u 1 show adapter
mpt1 Adapter:
       Board Name: SAS3801E
   Board Assembly: L3-01123-04E
        Chip Name: C1068E
    Chip Revision: UNUSED
      RAID Levels: none
The drive is seen by the adapter:
Code: 
# mptutil -u 1 show drives
mpt1 Physical Drives:
...
 da4 ( 3726G) ONLINE <SEAGATE ST4000NM0023 GE0F> SCSI-6 bus 0 id 7
...
 da6 ( 2795G) ONLINE <TOSHIBA MG03SCA300 DG02> SCSI-6 bus 0 id 27
...
 da9 ( 2048G) ONLINE <WDC WD221KFGX-68 0A83> SATA bus 0 id 19
...
smartctl can talk to the drive and interrogate it. Note that it's not 4Kn, but rather 512e (I was hoping it would report 4K logical and that would explain my problem):
Code: 
# smartctl -a /dev/da9
smartctl 7.3 2022-02-28 r5338 [FreeBSD 13.1-RELEASE amd64] (local build)
Copyright (C) 2002-22, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Device Model:     WDC WD221KFGX-68B9KN0
Serial Number:    2GH1YSAS
LU WWN Device Id: 5 000cca 2becefb90
Firmware Version: 83.00A83
User Capacity:    22,000,969,973,760 bytes [22.0 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    7200 rpm
Form Factor:      3.5 inches
Device is:        Not in smartctl database 7.3/5319
ATA Version is:   ACS-5 (minor revision not indicated)
SATA Version is:  SATA 3.5, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Tue Jun 27 17:53:26 2023 AEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
Here's where I end up with a 2TB drive:
Code: 
# gpart destroy da9
da9 destroyed
# gpart create -s gpt da9
da9 created
# gpart show da9
=>        40  4294967216  da9  GPT  (2.0T)
          40  4294967216       - free -  (2.0T)
Oddly, the drive does not show up at all on the shelf enclosure utility:
Code: 
# sesutil map | grep da
                Device Names: pass12,da10
                Device Names: pass6,da4
                Device Names: pass9,da7
                Device Names: pass10,da8
                Device Names: pass7,da5
                Device Names: pass4,da2
                Device Names: pass5,da3
                Device Names: pass8,da6
                Device Names: pass3,da1
So I'm guessing the shelf is the issue - I'm yet to try the drive directly connected to the controller, that's next.
Does anyone have any advice?
Thanks,
Scott
 
scotia said:
Code: 
da9 ( 2048G) ONLINE <WDC WD221KFGX-68 0A83> SATA bus 0 id 19
Click to expand...

scotia said:
Here's where I end up with a 2TB drive:
Click to expand...
It showed up as a 2TB drive by the controller above (2048G). Seems like an odd number, 2TB drives are rarely 2048 GB, more like 2000 GB. So something's up with the controller settings?
 
see mpt(4):
Code: 
WARNINGS
     Most controllers supported by the mpt driver suffer from limitations on
     supported disk size (mostly <2TB).  While most controllers will truncate
     usable disk size, others might behave unexpetedly and can cause severe
     data loss.  Refer to the datasheet of the chipset and firmware version
     for information about supported disk size and limitations.
 
And it looks like 2 SAS drives above have their size (that is > 2TB) correctly reported, so it could be related to the WDC one being SATA, you never know with these old controllers.
 
Just an update:
The drive works fine (shows as 22TB) on a MacbookPro using an external USB enclosure.
The drive works fine in the internal bay of an old HP MicroServer N39L.
On the FreeBSD 13.-1-RELEASE system in the OP I removed the shelf from the system and connected the drive directly to the LSI controller. Still shows as 2TB, confirming that the issue is the controller (not the shelf):

Code: 
# mptutil -u 1 show drives
mpt1 Physical Drives:
 da1 ( 2048G) ONLINE <WDC WD221KFGX-68 0A83> SATA bus 0 id 29

Code: 
# camcontrol devlist -v
scbus0 on ata0 bus 0:
<NECVMWar VMware IDE CDR00 1.00>   at scbus0 target 0 lun 0 (cd0,pass0)
<>                                 at scbus0 target -1 lun ffffffff ()
scbus1 on ata1 bus 0:
<>                                 at scbus1 target -1 lun ffffffff ()
scbus2 on mpt0 bus 0:
<VMware Virtual disk 2.0>          at scbus2 target 0 lun 0 (da0,pass1)
<>                                 at scbus2 target -1 lun ffffffff ()
scbus3 on mpt1 bus 0:
<ATA WDC WD221KFGX-68 0A83>        at scbus3 target 29 lun 0 (pass2,da1)
<>                                 at scbus3 target -1 lun ffffffff ()
scbus-1 on xpt0 bus 0:
<>                                 at scbus-1 target -1 lun ffffffff (xpt0)

And importing a ZFS pool already created in the drive failed (of course):

Code: 
zpool import
   pool: ZP-01
     id: 13885200093243380538
  state: UNAVAIL
status: The pool was last accessed by another system.
 action: The pool cannot be imported due to damaged devices or data.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-EY
 config:

        ZP-01       UNAVAIL  insufficient replicas
          da1       UNAVAIL  invalid label
 
If you read the link in my previous post you will see that your SAS3801E doesn't support disks larger than 2TB.
 
Just for info's sake - the controller works fine with 4TB, 3TB as well as 2TB drives. The 22TB is the first drive I've tried >4TB.

diizzy I'll check out the ASM1164 - thanks for the hint. Ideally I'll find an HBA that doesn't cause a Cisco UCS to ignore the power plan but that's another problem.
 
yuripv79 said:
That's not correct even looking at the original post, there are SAS drives bigger than 2TB attached to the same controller.
Click to expand...
one of the drives are SAS and other are SATA

Here's a quote from LSI regarding the SAS1 IT/IR Products and it's limitation
Hardware Limitation for SATA Disk Support: Hardware does not support greater
than 32-bit LBA addressing that is needed by READ/WRITE (16).

Software Limitations for SAS RAID Disk Support: Data structures in MPI interfaces
and IR firmware use 32-bit fields and 32-bit math operations. To change every impacted
field and the math operations to 64-bit arithmetic operations takes great effort and
requires fundamental changes to the board (firmware and software stack). The effort to
support such a change is not planned for the SAS1 products roadmap
Click to expand...

 
Yes that's why i provided the first link from Broadcom in my first post.

3Gb/s SATA+SAS HBAs will detect drives with greater than 2 TB in capacity (in other words, the 3Gb/s SATA+SAS HBAs will work with 3 TB SAS disk drives) if Phase 13 or newer IT firmware only (i.e. not IR firmware)(SAS disk drives only, not SATA) is used.
Click to expand...
 
scotia said:
Just for info's sake - the controller works fine with 4TB, 3TB as well as 2TB drives. The 22TB is the first drive I've tried >4TB.

diizzy I'll check out the ASM1164 - thanks for the hint. Ideally I'll find an HBA that doesn't cause a Cisco UCS to ignore the power plan but that's another problem.
Click to expand...

There's a difference between "shows the correct size" and "works with that size of drives". Have you filled any of those disks with more than 2TB data? Even LSI states you can't use more than 2TB with that HBA:

Using drives >2TB in capacity with LSI SAS HBAs

Using drives >2TB in capacity with LSI SAS HBAs
www.broadcom.com www.broadcom.com

They use some 16bit chips internally, which won't be able to address anything above that 2TB boundary. Been there, done that, lost a lot of data with some old MegaRaid controller (which had the same limitations) when the 3TB disks were filled >2TB and the controller just wrapped the address around and continued writing at the first sector...
Back then after a lot of troubleshooting I found a blog entry, where someone dissected the firmware and found that exact bug and the fact they used some 16bit controllers (IIRC with some weird truncated address space of 14 or 15 bit) which cause the limitation. That LSI FAQ entry didn't exist back then, you had to Learn The Hard Way™
 
Thanks sko that's good info. I think I'll bin this controller and try to find a another, more modern one with external interfaces. In a different machine I'm using a SAS9200-8e with the LSISAS2008 chip and it handles large drives fine (both cosmetically and in real use). What I would love to use is something that Cisco UCS chassis recognise so that I can get the fan speed down, but they need to have the Cisco VID/PID values...
 
I'd go at least to a SAS3008 / SAS9300 based controller. Those SAS2008 are also very old by now (PCIe gen 2!) and apart from quickly becoming a bottleneck, especially with SAS SSDs, they are much more power hungry than more recent controllers.
If this is a private build on a budget, I'd always go for a "white box" HBA except there is a *very* good offer for used controllers, preferably *not* some rebranded ones (e.g. IBM or DELL). Flashing those to vanilla IT firmware can be a PITA as they sometimes have some weird restrictions in place on what firmware can be flashed.
 
Just as a coda to this thread, I was doing some burn-in testing on some drives in the shelf using the SAS3801E, and even the 3TB SATA drives (that show correctly as 3TB) were truncated:
# dd if=/dev/zero of=/dev/da1 bs=16M
dd: /dev/da1: short write on character device
dd: /dev/da1: end of device
131072+0 records in
131071+1 records out
2199023254528 bytes transferred in 15868.464079 secs (138578204 bytes/sec)

which is exactly 2TB transferred before it failed.
So yes, even SATA drives that are correctly reported still can't be accessed beyond 2TB.
My SAS9300 is in the mail.
 
You must log in or register to reply here.
Back
Top