Other Swapped drives, lost activity LEDs

spork

spork

So this one has me stumped - on an old Supermicro 2U server with a hot-swap SATA/SAS backplane, I used to see drive activity reflected via the LEDs that show through the drive carrier "light pipes". I still do on two drives...

However the server has recently had a total of four drives swapped out. After each replacement, I lost the activity/status LEDs. The two older drives that remain still show activity via the LEDs. The server has been rebooted a few times and the situation has not changed - the old drives show activity, new ones do not.

Any ideas? Not clear if this is a function of the SES device or not. "sesutil map" doesn't seem to indicate it's involved (all slots show status "UNKNOWN", including the slots where drive LEDs are functioning properly).

Honestly not sure where to start, any ideas?

FreeBSD 11.2-RELEASE, all drives on the built-in ICH-10 AHCI controller.

Interesting dmesg snippets:
Code: 
ahci0: <Intel ICH10 AHCI SATA controller> port 0xa480-0xa487,0xb000-0xb003,0xac00-0xac07,0xa880-0xa883,0xa800-0xa81f mem 0xfbed6000-0xfbed67ff irq 19 at device 31.2 on pci0
ahci0: AHCI v1.20 with 6 3Gbps ports, Port Multiplier not supported
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich1: <AHCI channel> at channel 1 on ahci0
ahcich2: <AHCI channel> at channel 2 on ahci0
ahcich3: <AHCI channel> at channel 3 on ahci0
ahcich4: <AHCI channel> at channel 4 on ahci0
ahcich5: <AHCI channel> at channel 5 on ahci0
ahciem0: <AHCI enclosure management bridge> on ahci0

ses0 at ahciem0 bus 0 scbus7 target 0 lun 0
ses0: <AHCI SGPIO Enclosure 1.00 0001> SEMB S-E-S 2.00 device
ses0: SEMB SES Device

working activity LEDs:
ada4 at ahcich4 bus 0 scbus5 target 0 lun 0
ada4: <WDC WD4000FYYZ-01UL1B2 01.01K03> ATA8-ACS SATA 3.x device
ada4: Serial Number WD-WCC130YUZX0K
ada4: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada4: Command Queueing enabled
ada4: 3815447MB (7814037168 512 byte sectors)

not working activity LEDs:
ada2 at ahcich2 bus 0 scbus3 target 0 lun 0
ada2: <HGST HUS726040ALE610 APGNTD05> ACS-2 ATA SATA 3.x device
ada2: Serial Number K3H3K09L
ada2: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada2: Command Queueing enabled
ada2: 3815447MB (7814037168 512 byte sectors)

sesutil output:

Code: 
[root@trunk /var/tmp/vi.recover]# sesutil map
ses0:
    Enclosure Name: AHCI SGPIO Enclosure
    Enclosure ID:                0
    Element 0, Type: Array Device Slot
        Status: Unsupported (0x00 0x00 0x00 0x00)
    Element 1, Type: Array Device Slot
        Status: Unknown (0x06 0x00 0x00 0x00)
        Description: SLOT 000
    Element 2, Type: Array Device Slot
        Status: Unknown (0x06 0x00 0x00 0x00)
        Description: SLOT 001
    Element 3, Type: Array Device Slot
        Status: Unknown (0x06 0x00 0x00 0x00)
        Description: SLOT 002
    Element 4, Type: Array Device Slot
        Status: Unknown (0x06 0x00 0x00 0x00)
        Description: SLOT 003
    Element 5, Type: Array Device Slot
        Status: Unknown (0x06 0x00 0x00 0x00)
        Description: SLOT 004
    Element 6, Type: Array Device Slot
        Status: Unknown (0x06 0x00 0x00 0x00)
        Description: SLOT 005
 
Normally, the per-disk activity LEDs are a function of the disk itself: The SATA and SAS connector has a pin for driving the activity LED, and the disk is what controls the LED.

What could have gone wrong here? Good question. Mechanical or electrical problems? Also, at least on SCSI (SAS) drives, you can program the drives via mode commands whether to control the LED at all, and whether the state "power on and inactive" is shown as solidly on or solidly off. Also, changing the firmware on the drives can affect how the drive controls the LED.

Your message is not 100% clear: Is the lack of LED exactly correlated with new versus old drives? It could be that the new drives are not driving the LED. Find out what model they are, and get the full documentation (the SCSI or ATA manual) for them, and figure out whether it is possible to control their behavior. This might also require a phone call to support from the disk drive vendor.

Now it is theoretically possible that the SAS expander = SES device that's on your disk backplane could control the LEDs. I've never heard of a SAS expander that does that, but new things happen all the time. Here is how you can find out: Get a good magnifying glass, find the LED on the backplane, and look for the copper trace that connects the LED. One side will go to +5V or ground, boring. The other side will either go to the SAS expander chip or to the SAS/SATA connector (perhaps via a resistor).

Or call Supermicro support and ask them.
 
These are relatively modern SATA drives. They have no jumpers nor any connectors (LED, Master/Slave, etc.) aside from SATA and power (nor did the older drives that are still indicating activity on the SES LEDs). The only connection between any of these drives and the backplane are the power and SATA connectors. I think I'd have to go back to the '90's or maybe the early part of this century to find drives that had to have an actual cable attached from the drive to the activity LED. I don't really think this is mechanical/electrical, this has to have something to do with how the kernel/driver gets bus activity and then relays it to the SES backplane or whatever it is that's controlling the LEDs - much like a desktop rig - the LED for drive activity is not connected to the drive, but to a connector on the motherboard that's driven by the onboard SATA controller.
 
The connector for SAS and SATA power on the drive has an LED connection; I just looked it up, and it is pin 11 of the connector. That's how the disk drive itself can control the LED on the backplane. See this document from T10 (the SCSI committee), and scroll to page 102. This is how disk drives commonly control the activity LED.

Now, the LEDs on your backplane may not be connected to the drives; they could be connected to the SATA controller, or to the data switch (the thing I called the "SAS expander") on your backplane. Only Supermicro knows for sure.
 
Related, this is apparently the backplane in use: https://www.thomas-krenn.com/en/wiki/BPN-SAS-823TQ_SAS/SATA_Backplane

I'm reading the very long drive datasheet here to see if there's anything about pin 11 other than the fact that it has two uses (drive activity and staggered spin-up):

https://www.hgst.com/sites/default/...star_7K6000_SATA_OEM_Specification_Rev1.3.pdf

This is totally unrelated, but an interesting read:

https://www.hgst.com/sites/default/files/resources/HGST-Power-Disable-Pin-TB.pdf

In some HGST SATA drives, they've changed pin 3 to be a "hard reset" toggle. Put 3.3V on that pin and you power-off the drive. Release and the drive powers up. This is apparently a SAS feature to "reboot" a hung drive (yikes, maybe drive firmware is getting too complicated).

I'm opening a case with HGST just to see what they say about whether or not there's some toggle to send activity blinky-blinks on pin
 
Are the new hard-drives comes with they own hot-plug tray or you are using the tray from the old hard disk?
I've seen some replacement that mismatch the plastic pipes to the back LEDs or entire missing pipes and even if the light is working it's not visible to the front on the tray.
If you still have the old tray you can try to swap the hdd only to see if it help.
 
VladiBG said:
Are the new hard-drives comes with they own hot-plug tray or you are using the tray from the old hard disk?
I've seen some replacement that mismatch the plastic pipes to the back LEDs or entire missing pipes and even if the light is working it's not visible to the front on the tray.
If you still have the old tray you can try to swap the hdd only to see if it help.
Click to expand...

Old trays - just pulled the drives, unscrewed, put new drives in the same sleds.

I have email out to Supermicro (doubt I'll hear anything, their support page suggests emailing a reseller that doesn't exist anymore) and to HGST. Supermicro I'm asking about what method that super-server uses by default for drive activity and if there are other options, HGST I'm asking if they can confirm pin 11 does what it says it does.

That said, this is only a 6-bay server, I've already confirmed the drive cabling is sane enough that all the drives are in order (ada0 top left, ada5 bottom right), and recorded the serials with drive position so I can find them all in the future.
 
I take it back: This backplane doesn't have a SAS expander chip. Instead, it has a dedicated SES chip, which connects via I2C or SGPIO to the disk controller. And the backplane has a dedicated connector for driving some LEDs.

The question that arises is this: How many LEDs are there? I suspect that each disk slot (and each carrier) has two LEDs, and two light pipes. Probably one LED is green or blue, and is used for "disk activity", and controlled by the disk drive itself. The other LED is probably red or yellow, and is used for "find me" or "service alert" or "this disk is sick", and is driven by the HBA (also known as the disk controller). The manual is not clear about that, since it only describes the connectors for the LEDs, which seem to have many different options.
 
You must log in or register to reply here.
Back
Top