ZFS Don't ever buy from ACME hardware

gkontos said:
But the issue here so far has been the fact that we don't get any support. The response "try to set up the free software that is not many people used" is strictly unprofessional.
Click to expand...

I think "unprofessional" is a very friendly word in that case.
 
You probably want to upgrade the firmware if you haven't as it might fix your issue, in my experience using the newest firmware available is 98% always a good idea at also includes BIOS versions of motherboards.
As other's have mentioned you probably want to check cables and backplane along with the controller itself but this being a remote server might make it a bit hard.

I don't have any solid statistics but I do see occasional quirks when using WD's higher end SATA drives even on Intel controllers and I would guess the LSI ones are pickier in general. I know Toshiba consumer drives plays nice with LSI controllers in general but those are only 4TB (ACA-series). HGST drives used to be good ones but since HGST is owned by WD now-days I don't know if the new 5+TB drives are based on the old IBM design like Toshiba or if they are pretty much rebaged WD drives.

The Storage Forum over at HardForum might be of interest as it covers similar setups and have pretty good traffic.
//Danne
 
The firmware on the card has been updated to the latest version + bios. They also sent me a new firmware to flash the SAS expanders which actually made the system unbootable. Their latest response is:

Hi George,

We had tested the hardware, fixed the backplane issue. We had tested 3 different hard drives, so far the ZFS pool function still not work reliable with the LSI SAS3 controller + SAS3 expander configuration. Our Linux expert is looking into the software problem and solution.

Do you must have the ZFS for your applications?

Can you use hardware or software RAID for the storage space?

The other way is try not use the SAS3 expander, just direct connect HDDs to controller. But that will limit the # of hard drive (if the 8-port controller can just control 8 drives).

That is the result now. What is your idea?

Thanks

Nick
Click to expand...

We are at the point now of getting our money back from them and buying an new server from IX Systems.
 
I honestly don't think that it's "their" fault, you're probably looking at some hardware issue/incompatibility and expanders in general seems to be a hassle from what I've seen on Hardforum with LSI cards. Looking at the very sparse information iXsystems provides it looks like they're not using expanders at all on their systems and wire each drive directly to a port so I would suggest trying that.

The only expander I know that seems to work reliably (I haven't done much research in this area) is one by HP (SAS2)
http://hardforum.com/showthread.php?t=1484614

You could also drive enforcing slower speeds such as SAS2(6G) in controller firmware if that helps with the current hardware.

So I would go this route (preferably),

1. Force slower neg speed (6G)
2. Exclude the expander --> Wire directly
3. Try another controller card --> Same model
4. Try another controller card --> Previous generation
5. Try another brand (preferably something else than WD or Seagate unless it's SAS) of HDDs (wired to the card)
6. Accept the fact that you have a fun hardware/software issue :/

//Danne
 
diizzy said:
I honestly don't think that it's "their" fault, you're probably looking at some hardware issue/incompatibility and expanders in general seems to be a hassle from what I've seen on Hardforum with LSI cards.
Click to expand...

Honestly, you have no idea regarding the type of responses that we got from those guys. If there is an incompatibility then SuperMicro needs to be involved and provide a solution with a different chassis or controller. We have build many systems using SuperMicro JBOD's with expanders without having any issues what so ever. (Of course, dealing directly with SuperMicro in Europe)

diizzy said:
Looking at the very sparse information iXsystems provides it looks like they're not using expanders at all on their systems and wire each drive directly to a port so I would suggest trying that.
Click to expand...

No you are wrong, you can't wire 36 drives to a controller. Also, their configuration that they provided is build to order, pretty much similar so what we have. The only difference is that they guarantee that their system will work with FreeBSD 10.1-RELEASE.
 
diizzy said:
So I would go this route (preferably),

1. Force slower neg speed (6G)
2. Exclude the expander --> Wire directly
3. Try another controller card --> Same model
4. Try another controller card --> Previous generation
5. Try another brand (preferably something else than WD or Seagate unless it's SAS) of HDDs (wired to the card)
6. Accept the fact that you have a fun hardware/software issue :/

//Danne
Click to expand...

Since you edited this, I am replying again:

1) The controller card has been replaced. (Please read the full thread)
2) The server returned to their premises in order for them to try all the combinations that you mentioned.

Responding, do not use ZFS, use a different FS is like saying, IGNORE your data integrity. I could use ZFS without checksums, right? Would that solve my problem??
 
I see, no you can't wire 36 HDDs to a single card but you can use multiple controllers. LSI have up to 16 drives per card and if need use the builtin Intel controller for additional drives. Without proper data sheets I don't know how iXsystems configure their boxes but Netflix uses several controllers and hardwire everything so likely they've gone the expander route and found that performance and/or reliability wasn't satisfactory.
https://openconnect.itp.netflix.com/hardware/index.html
Interestingly they seem to have stayed on SAS2 hardware intead of going SAS3...


I did find the "use a different FS" a funny comment, but yeah... you're looking at some hardware/driver related issue probably as you get the same issue on both Linux and BSD. I do however understand the comment you got by using all "new" hardware that are not widely used (in general) and possibly mixing with something else than Windows or possibly Linux. I'm not saying FreeBSD is bad but getting "new" hardware to work reliably in FreeBSD often takes quite a bit longer than on Windows or Linux in general. It does however doesn't seem to the be case as you get the same results in Linux.
//Danne
 
Danne, please go ahead and read one of their messages on Tue, 10 Feb 2015 13:58:07

Hi George,

Sorry for the late reply. Supermicro Lab is still trying to narrow down the root cause of the problem. Originally there was a problem with the backplane, and that is fixed now, but there is still a problem with the data corruption. So far they have tried it on a smaller scale, with just 4 HDD and 1 cache, but the problem still occurs.

I will update to you once I have any further information.
Thank you and have a nice day!
Selene
ACME Micro Systems
Click to expand...

Now, after reading this. Would you trust your data there? Please do keep in mind that the tests have been supposedly conducted on both FreeBSD and CentOS 6.6
 
I've never been much of a fan of Supermicro as they seem overrated to me but some swear by them. Anyways I don't know if it's an issue with Supermicro's hardware design or it's caused by LSI's controller. As far as I know pretty much all rebrands of LSI cards use their reference design (Supermicro used to at least flip the PCIe connector on their UIO cards). I do however think that you might have been bitten by the early adopter of SAS3. You're going to have a hard time finding a vendor offering that setup in a single 4U case without it being a pure storage device. I have good experience (although limited) with Fujitsu myself and they work fine with FreeBSD but they do not have have that large servers, they seem to max out at 12 HDDs but at least according to their datasheets they seem to offer SAS3 controllers by default (Fujitsu do offer SAS2 cards too). iXsystems also use Supermicro for with its worth...
//Danne
 
After a lot of BS from ACME we decided to pay the 25% restocking fee and buy the system from IX Systems.

Thank you
 
diizzy said:
So... They didn't have an older LSI card available?
//Danne
Click to expand...

The LSI3008 is onboard.

I wanted to buy the same board in a SM 2HE Case with 16x SAS3 Backplane using SATA drives.
But after I read this thread again
and another thread
https://forums.freebsd.org/threads/what-happened-to-vfs-zfs-vdev-max_pending.50591
and this old opinion sadly still seems to be true
http://garrett.damore.org/2010/08/why-sas-sata-is-not-such-great-idea.html

I am not sure anymore if it is a good idea to use SATA drives in SAS environment.

Gkontos, has the IX System already arrived? Is it working?
 
User23 said:
Gkontos, has the IX System already arrived? Is it working?
Click to expand...

We are expecting it. It will ship next Tuesday or Wednesday and I hope it will arrive in the DC by next Friday.

I know that using SATA drives is not the best idea. I have a friend who is building large storages. Very large storages! He is using only SATA drives. As a matter of fact he even uses the cheap WD greens on some builds. He never had any issues.... Call it luck??
 
gkontos said:
I know that using SATA drives is not the best idea. I have a friend who is building large storages. Very large storages! He is using only SATA drives. As a matter of fact he even uses the cheap WD greens on some builds. He never had any issues.... Call it luck??
Click to expand...

Iam using a old CSE-846E2-R900B filled with WDC SATA WD1002FBYS over BPN-SAS-846EL2 Backplane connected to a 3ware 9690 SAS Controller.
No problems so far with the SAS connected SATA drives, beside the fact that it is old and need to be replaced soon.

Thats the reason why I am a little bit unsure if I have to build a server with direct attached drives, so i can be sure it is working rock stable "even" with SATA drives. Did your friend already use 12GBit SAS hardware? I would prefer the 12GBit SAS solution, but I would rather go back to 6GBit than building directly attached drive bays with 32+ SATA Cables on 4 different controllers.
 
I will ask him and let you know. I know that he uses SAS expanders with SATA drives and LSI HBA controller. I can't recall which model now.

If you want I can also give you the specs of the new system that we bought from IX Systems.

Screen Shot 2015-03-06 at 5.05.15 PM.png
 
robroy,

Her name is spelled and pronounced very differently... (I am not going to mention it in a public forum)

While it is true that the salesperson can not provide any support, they ought to be able to refer you at least to the manufacturer. In their case this never happened. Instead they claimed to work together with the manufacturer in order to solve the problem but it was clear from their questions that the person who was asking them had no idea about *nix...

I am not going to accuse in public someone UNLESS I have SOLID evidence for it. To anyone who is interested I have kept tons of email exchanges.

Buying hardware from them was not my choice. My client decided to and he regrets the day....

I am not also trying to sponsor IX Systems, and rest assure that if something is wrong I will be the first to mention it.
 
User23 said:
I would prefer the 12GBit SAS solution, but I would rather go back to 6GBit than building directly attached drive bays with 32+ SATA Cables on 4 different controllers.
Click to expand...
It isn't always that bad - I built my RAIDzilla II servers with 16-port SATA controllers which have SFF-8087 connectors, so there are only 4 data cables to the backplane, plus one I2C cable for the 3 status LEDs per drive slot. Here is a picture.
 
Terry_Kennedy said:
It isn't always that bad - I built my RAIDzilla II servers with 16-port SATA controllers which have SFF-8087 connectors, so there are only 4 data cables to the backplane, plus one I2C cable for the 3 status LEDs per drive slot. Here is a picture.
Click to expand...

Nice project, it feels like I saw it before. :)
I am surprised about the detailed informations ci design provided in the NSR316 manual and additional documents.
Unfortunately the newer CI Speed 16 or 24 cases have only pictures und general tech specs.
I will see if the european reseller can provide some more information.
 
gkontos said:
I will ask him and let you know. I know that he uses SAS expanders with SATA drives and LSI HBA controller. I can't recall which model now. If you want I can also give you the specs of the new system that we bought from IX Systems.
Click to expand...

Thank you!
I would eat every information about working FreeBSD LSI HBA -> Backplane -> SATA drive combinations I can get.
 
I really feel like a super idiot. I lost my client and damaged my business. BUT this is my fault. I trusted IX SYSTEMS!!!! I thought that buying a "certified" build according to their experience would work.

How little did I know....

The system arrived and:

1.png


The send a new controller however the situation is the same. My client spent an extra $23K

I am heading towards a nice whisky store....

It was nice meeting with all you people here, I will miss you.......
 
Sorry to hear :/

This seems related to http://zfsguru.com/forum/zfsgurusupport/343
Are you running this inside ESXi or some other kind of virtualization? It does look a bit odd that it only seems to occur on mps1 which is the second controller although I have no idea if its the builtin one or the external card but as you mentioned I guess this would be the PCIe HBA card. I do have two HBA's laying around but unfortunately my testbox doesn't allow me to run two controllers at the same time (it only has one >8x PCIe slot).
I don't see how iX missed this really as it seems to occur pretty much as soon as the machine boots up?
//Danne
 
You must log in or register to reply here.
Back
Top