Sterry.Me.UK      Living a Simple Life
      - don't make plans -

Archive for November, 2009

Nov
18

Around 18 months ago, I got myself a cheap Dell workstation on eBay – a dual Xeon 2.4GHz system, with 8GB RAM and upgraded to 16GB. It does have a lot of hard drives spinning at 15,000 rpm. They spin twice as fast as most desktop hard drives and are enterprise grade.

However, the Dell case, a Precision T5400 case, cannot hold more than four hard drives and I used another case to house the 12 hard drives and a bunch of DVD drives.

This presented another problem – cables were needed to connect the computer to the hard drive case. And these are not ordinary cables. The RAID controller card, an Adaptec 31605 has four unified SAS/SATA ports – each with four lanes – giving a total of 16 lanes – ie, 16 directly attached hard drives (more hard drives can be added than this to the controller by the use of an SAS expander, but that’s a budget I can’t justify). So the data cables have to have these kind of plugs:

Span.com - SFF8087 miniSAS-36 to SFF8087 miniSAS-36

The image below shows the data expensive data cables coming out the front to the case below:

Dell Precision T5400 exposed

Now these ports do not have a normal home style SATA type socket – there are many more variations on the SAS/SATA sockets than what is seen in the home computers – this rules out long SAS/SATA cables that you can get for about 2 quid on eBay. I needed a 2 metre four-lane cable with specialist male external plugs on either end. And four of these.

4-lane SAS cable - miniSAS-26 to miniSAS-26 SFF8088 to SFF8088

The price for each of these were 40 quid, so its expensive stuff. Add to that bill are two four female sockets to accept the cable and convert to a 4-lane internal socket.

Span.com 4 port internal/external sockets - miniSAS SFF8087 to SFF8088 - PCI Bracket

From there it finally split off into four cables each to the hard drives.

SAS hard drives do have similar power and data connectors as SATA drives – with one annoying caveat – there is no break between the power and data ports on the SAS – it form one long continuous connector – so you can’t use the individual power and data cables like you do on home SATA drives.

Span.com SAS hard drive cables -Multilane - SFF8484 SAS-32 to SFF8484 SAS-32

The DVD drives were simply connected by 2.5 metre SATA cables straight off the motherboard and out through into the hard drive case.

The hard drive case also need to be powered separately.

This makes for a rather untidy and inelegant setup. So – I’ve long pondered transplanting the Dell motherboard into another, larger case that could accommodate all the hard drives and DVD drives.

Inelegant computer set-upстолове

1) It would mean all expensive cables are now redundant (although if I use more hard drives in future, then these would come in handy again).

2) Dell computers are famous for their non-propriety parts. I was concerned that while the motherboard looks like an EATX server board standard, the screw mounts may be offset ever so slightly. Their power supplies were famously non-standard – the plug would fit but the wiring were switched, leaving one with a fried motherboard. I’m told this was in the old days but it certainly fills me with confidence.

3) I wanted a case that was not too tall, to fit on the shelf in the cupboard which had a low sloping ceiling. It needed to support two power units as well.

In the end, I went for the Lian Li V2010B. It supports EATX sized boards, two power supply units. I believe Lian Li no longer makes these and there are newer versions perhaps.

Lian Li V2010B

It took around 45mins to remove the Dell motherboard completely from the Dell case and install it onto the removable motherboard tray as shown below:

Dell Precision T5400 mounted on a Lian Li EATX motherboard tray

It took another 15 mins to mount it inside the case along with the Dell PSU. I chose not to buy a third-party PSU yet as I’m unsure about the second power connector on the Dell motherboard – I know I’d need to make sure I get a third party PSU that is EATX compatible at least. I’d like to so I don’t have to use two power switches.

Dell Precision T5400 transplanted into a new case

Immediately two problems became apparent as I began to add the bits and pieces

- the heatsinks for the CPUs will protrude into the Lian Li CPU cooler fan. Luckily the fan is quite adjustable.

- the PCIe/PCIx slots on the Dell board do not line up precisely to the Lian Li PCI openings. Only two of the lowest slots will fit, but as you go up the slots, it becomes more and more out of alignment – which is annoying as the higher slots are where my graphics card and RAID cards belong and they need to be well supported.

I have sort of bodged it by placing some thumb screws into the PCI brackets below those cards to support the cards. The top level card, which is a PCIe x8 slot wired as x4 and is used the the PCIe x1 wifi card and is completly out of alignment with the Lian Li PCI openings. Fortunately, the wifi card is light enough to be self supporting in its slot as long the external antenna cable isn’t pulled in the wrong direction!

The PSUs I have, both the Dell and the weird no-brand XPOWER PSU from Maplins has screw mounts that do not line up with the Lian Li PSU screw mounts – I could only get two screws in for both. Not ideal for such heavy things.

The 3.5″ hard drives, all are noisy 15,000 rpm SAS drives had to be taken from their 5.12″ to 3.5″ mounting brackets and put on Lian Li screws with rubber grommets. When the drives are slid into the case’s drive bays, it actually use those rubber grommets as the support points, not the screws themselves.

I wasn’t convinced those rubber grommets will do much good in dampening the 15krpm drives’ vibrations and noise. Furthermore, modern hard drives shed their heat by thermal contacts with the case mountings. And 15,000rpm drives do get very hot, probably hotter than those 2TB drives. However when in the case, I was amazed that the noise and vibrations were substantially reduced – those tiny grommets worked wonders. And my drives were cooler in this Lian Li case than in the old hard drive case with traditional mountings. This is in no doubt due to the dedicated 120mm fan blowing cool air over these drives and the resulting hot air sucked out by the PSUs’ fans on the other side. It’s a testament to Lian Li’s superb airflow design.

The SuperMicro 2.5

In fact, the 4 hard drive 2.5″ single 5.12″ backplane by SuperMicro is the nosiest component that can be heard from outside the case no thanks to its screeching tiny fan that runs at full speed irrespective of conditions. This isn’t the manufacturer’s fault as it was never designed for the home environment, it is usually for those noisy server rooms.

That bloody fan!

Wiring up 8 hard drives, and four DVD drives and a further 4 2.5″ drive backplane required careful planning if you want to keep it neat and tidy. That’s 13 data cables (the backplane uses one 4-channel data cable) and 13 power cables. It is made all the more worse when the 3.5″ SAS drives use a splitter to get its power through Molex sockets – damn inelegant! An example of it below:

Span.com SAS and power connectors

In the end, I managed to get it as tidy as this with the help of some braiding to help with the cable management and improve airflow.

Inside the new Lian Li V2010B case with the Dell Precion T5400 motherboard

And isn’t it one very classy looking Dell Workstation?

Dell Precision T5400 inside Lian Li V2010B case

And the desktop in the other room…

Desktop with Dell 3008WFP

So to sum up – it was a relatively straight forward physical transplant of the Dell Precision T5400 EATX server board to a third party case with the following caveats:

1) The front panel with two USB sockets, audio sockets and the power on/off/reset switch is Dell’s properiety standard and have not figured out the pin layout except for the power on/off pins. Dell do not publish these and no one on the internet has done any homework yet!

2) The PCIe/PCIx slots may not line up with the PCI brackets on the case

3) The power supply for the board – I have not yet tried a third party PSU and kept to the Dell PSU for now and the screw mounting do not line up with Lian Li’s case screw mounts.

All the specialist SAS/SATA cables, ports and backplanes were brought from www.span.com – an IT company based in Surbition in South West London, UK – see bottom of page for full list of components and links should any of you are crazy enough to try it out!

Computer specification:

Dell T5400 Precision workstation
Lian Li V2010B extra large EATX tower chassis case
2.4GHz E5410 Intel Xeon processors (x2)
16GB FB-DIMMS, 666MHz
2GB Palit ATI Radeon 4870 OC hard with non-reference cooler PCIe x16 card
Adaptec 31605 16-port RAID PCIe x8 card with 256MB cache and battery backup
Wifi PCIe x1 card

Storage:

15,000 RPM SAS hard drives, 3.5″ (IBM, HP, Hitachi) (x8)
15,000 RPM SAS hard drives, 2.5″ (HP) (x4) (soon to be x8)
4-hard drive 2.5″ in 5.12″ backplane, SuperMicro x1 (another one to come)
Blu-ray rewriter and HD-DVD reader drive, LG, SATAII
DVD-Writer drives, Dell, x3

PSU:
Dell Precision 850 watts PSU
XPOWER 600 watts PSU
Total: 1450 watts

UPS:
Belkin 1200 kVA unit

Monitor:
Dell 3008WFP 30″ 2500×1600 connected via DisplayPort / alternatively via Dual DVI

OS:
Windows XP 64bit (considering going Windows 7 64bit Professional)


    Span.com shopping list:


In final single case:

2x SFF8087 (miniSAS-36) Controller to 4x SFF8482 (SAS) Device, 75cm internal SAS/SATA, Multilane fanout [From RAID card direct to SAS 3.5" drives]
1x SAS/SATA Internal Backplane, CSE-M14TB, 4x 2.5″HD SAS, 1x 5.25″Bay [2.5" SAS backplane]
1x (miniSAS-36) to SFF8484 (SAS-32), 50cm internal SAS/SATA, Multilane [From RAID card to 2.5" SAS backplane]


If having a two-case setup, below is required:
(Remember each cable carries four SAS/SATA signals)

1x Standard 300W PSU; uncabled, DA-18, 18-Bay 5.25″ Case [Case is designed for DVD/CD Drives, may require changing 300W PSU for larger wattage if you have lots of power hungry hard drives like me - read hard drive specifications and add plenty of watts on top per drive for headrooms (eg, for spin-ups) and PSU with enough Molex/SATA plugs]

12x 5.25″ to 3.5″ mounting brackets – get them from eBay Search Results in this link

2x miniSAS (SFF8087 to SFF8088) – PCI Bracket, 4 port SAS/SATA backplate (internal to external/vice versa), 4 multilane ports – for Areca cards, use another internal/external backplate card on span.com

3x miniSAS-26 to miniSAS-26 (SFF8088 to SFF8088), SAS/SATA external cable, 3m, Multilane [From one internal/external backplate to another]

3x SFF8087 (miniSAS-36) to SFF8087 (miniSAS-36), 30cm internal SAS/SATA, Multilane [From internal/external backplate to 3.5" hard drives]

3x SFF8087 (miniSAS-36) Controller to 4x SFF8482 (SAS) Device, 75cm internal SAS/SATA, Multilane fanout [From backplate direct to SAS 3.5" drives]


Other things handy to know:

If you have a separate case for hard drives, DVD/CD drives, and you buy a separate PSU, chances are, it won’t power up because you haven’t plugged in the redundant ATX motherboard connector. So, you can wire up two pins on the ATX plug to a switch or simply join them together. See http://www.directron.com/2powersupplies.html#addatx - but beware of old propriety Dell ATX connectors!!! Also, do not power on without at least one device draining electricity from the PSU or the PSU will burn out if nothing is connected and is powered on. Do this at own risk, do some research and if in doubt, leave it, I am not responsible for you making a mistake.

ATX connector - manual power on

It is difficult to get hold of internal SATA cables longer than 2m – because SATA is not certified for longer lengths. 2m ones can be found on this eBay search link If you want to go longer than this, then get some cheap SATA to eSATA PCI brackets and you can get long eSATA cables (possible to get 2 ports per brackets). I don’t fancy doing this for data critical SATA hard drives, but DVDs/CDs/Blu-Rays seem fine.

Having more than one PSU is typically better than one very high wattage PSU due to efficiencies. This is because it is difficult to find a very efficient high wattage PSU (this is by limitations of technology and design) while you can get an 80 Plus Certified PSU, look closer and you can see the efficiency is less than 80% at maximum output. For example, 1000w is burned just to give you 750w. And since you are using just one PSU, you’re going to work that PSU harder – ie, need more watts out of it to power everything, potentially close to its maximum ratings where the efficiencies tail off, more heat and more fan noise. Two lower wattage PSUs can share the load, and because the load is shared, they aren’t working at their maximum draw and reach that nice apex in the PSU’s efficiency curve.

It is far easier to manufacture a 500w PSU that can reach as high as 87% efficiency, and probably higher still at slightly less than maximum draw. For example, a 500w PSU could have 90% efficiency at 480w and gives you 432w – two PSUs means 960w being burned to give you 864w instead of one PSU burning 1000w to give you just 750w.

The above is a crude example to illustrate how two PSUs can be better than one if you have high power draws – so it is worth doing your research on how much power you use and what PSUs are available on the market – not for greenie points but simply to make your PSU last longer, less heat to worry about, less noise and gives your backup UPS more battery time when the power goes out!