http://log.momentics.ru/homemade-48tb-enterprise-storage-system

I’d ask the community to review the article and probably to speed its writing up by asking questions… by twit or mail.
However the article is not complete, so I’d expect to fill gaps by answering questions.

btw: this all is written in Russian but contains lots of photos and graphs, so probably might be helpful

Yours, momentics

Its something I think we all face as geeks, but I can’t seem to find the ’sweetspot’ in relation to costs/performance. I want to obviously make this as power efficient as possible, but still retain the performance in regards to parity re-generation etc.

I have a few Build options listed here:

http://forums.overclockers.com.au/showthread.php?t=908505

If you had the time to read and pass any comments. I had a look into Atom, and believe the performance from them will be too lackluster. That then leaves me with looking at i5/910e (can’t for the life of me find them in Australia), or to the higher end Xeon/AMD X6.

Thoughts/Suggestions?

Good luck building that system. Just took a look at the Norco RPC-4224, and it is quite a beast with 24 drive capacity utilising six SFF-8087 mini SAS connectors. Three Supermicro SATA/SAS adapters will handle all those drives if fully populated. It should make a really good video storage NAS with massive capacity (upto 48TB raw capacity using the currently-available 2TB drives).

Regarding the fact that the Supermicro cards use a proprietary UIO backplate, you can either just unscrew the backplate and leave it off, relying on the fact that the mobo is horizontal in the RPC-4224 case, so the cards will normally be fine as they will be vertically mounted into the motherboard, or for more peace of mind, you will probably want to get a PCI 2.0 backplate from a component store and then reverse the screw mountings to fit the card.

Cheers,
Simon

Came across your thread today and gave me some great insight as I am currently constructing a Fileserver/HTPC as we speak.

The problem I am facing is that I have now just discovered (thanks to your blog) that the AOC-USASLP-L8e have their components on the opposite sides to most. I am using (or plan to) a Norco 4224, and was wondering how difficult this would be in a case like this.

I do not have any background in electronics/soldering and was wondering how difficult it would be to remove the backplate?

Cheers,
Trident911

Thanks for the compliments!

I haven’t really followed the situation closely regarding Oracle and OpenSolaris, but it does seem a great pity that they have dropped OpenSolaris, even if they are supporting Solaris. I will have to read more regarding the situation about Solaris to be able to answer your question properly.

Earlier in 2010 I upgraded both my OpenSolaris systems (NAS & clone) to build 134, and from what I read earlier it seems that it was build 134 that was destined to become OpenSolaris 2010.03 or whatever they were going to call it.

I can say from my own personal experience that build 134 seems to have been very good and I have not experienced any serious bugs with this build, so I can recommend its use if you want to consider using it.

For the meantime, until I discover what Oracle’s position is on home users using Solaris (i.e. will they charge & will they allow software updates?), I will continue to use build 134.

However, if Oracle chooses to charge home users/non-profit organisations for using Solaris, even if only for running a home NAS, then I will definitely consider using other operating systems, such as FreeBSD, which also has a ZFS implementation that was ported from the Sun code. This would be a great pity, as I would rather know I am using the reference implementation of ZFS from Sun/Oracle, but that’s a decision I will have to make later, depending on what I discover.

I heard somewhere that some developers have/will fork the OpenSolaris code, but I haven’t looked into this much yet. I believe the OS name is Illumos — see http://www.illumos.org/ . If this really happens and has proper developer support, it might turn out to be a good alternative to Solaris, assumuming that Oracle want to try charging home users for running Solaris.

Also, there is OpenIndiana, another fork of the OpenSolaris code. I’m unsure of the exact relationship with Illumos, but you might find the following links interesting:

OpenIndiana site
OpenIndiana .iso files for each of their releases
OpenIndiana downloads page (including instructions for upgrading from OpenSolaris)
OpenIndiana twitter stream

I hope this answers your points reasonably well.

Cheers,
Simon

I have to agree with other who posts in here, brilliant site and source for inspiration! Actually to such an extent that I during the spring decided to go OpenSolaris for my home file storage set up – mainly due to the data integrity provided (read ZFS).

I paused my project during the spring to wait for the next release to come 2010.3 … which never came and will never come – since OpenSolaris now is buried by Oracle.

My question to you is now what your thoughts are and what you would do in my shoes – like so many others.

Would you consider setting up a storage server today based on the last OpenSolaris release or rather chose another *nix platform?

I will mainly use the server for as “safe” file storage as possible in the sense of up-time and data integrity in mind.

Will the current state of OpenSolaris and ZFS be sufficient and “safe” – having the fact that the project is dead and the community might diminish with the years to come?

What are your thoughts and advice for people like me, who right now hesitate to take the route with OpenSolaris (or the Solaris 11 Express to come)?

Would appreciate your input and advice.

Cheers!

// Pelle

>What do you mean by that? Zpool scrub killed your drive(s)?

More of a clay pidgeon shoot.

If Fault Management sees too high an io error rate on disks during the scrub, and drops them out of the zpool.

When there are insufficient working drives left, the zpool goes offline.

A reboot “may” bring it back, but the inability to scrub to fix or detect errors leads to slow decay and eventual failure and bye bye to 30Tb of data.

It’s difficult to recommend something specifically without having tried it personally, but I would have thought an AM3 mobo with an AMD Athlon II X2 ‘e’ low power processor should be fine. The main thing for the mobo is to check out the chipsets to ensure they have good driver support – try looking on the OpenSolaris HCL.

The mobos I used are old now: (1) Asus M2N-SLI Deluxe (NAS) and (2) M2N-E (backup server).

If you get one, let me know how that ‘Fractal Design Define R2 Black Pearl’ case turns out.

Cheers,
Simon

I’m currently shopping around for possible hardware. If you have any recommendations for current AMD processors/motherboards, I’d appreciate any advice. Obviously ECC memory and low power use. It seems rather hard to find a mobo like what you recommend that is verifiable in being supported by opensolaris. I did find a dealer to buy the mobo you are using, but below is a possible option.

Mobo I would consider if it works out of the box:
ASUS M2N68-AM PLUS: http://www.newegg.com/Product/Product.aspx?Item=N82E16813131613

Ironically, I keep finding Micro ATX mobos to put in a rather large Fractal Design Define R2 case made in Sweden.

The old server was based on a Fujitsu-Siemens Activy 300 HTPC. It had a Celeron CPU with ~700 MHz, 256 MB RAM, two 160 GB Samsung disks. This was in between getting a little tight diskspace-wise. I was already pondering to replace it, as it had been running for over 4 years, with the hardware except the disks even a bit older. When the machine once stopped to work suddenly, and came back only at the second hard reboot, I began with the replacement in earnest.

The new server is built around a Zotac mini-ITX board with NVIDIA ION chipset and Atom 330 CPU. I used a Sharkoon Rebel 9 case, which has space for up to 9 disks, and (for the beginning) 4 EcoGreen Samsung disks with 1.5 TB each. 4 GB RAM (the supported maximum) is okay, although not really abundant. No ECC support, unfortunately. OS is OpenSolaris snv_132, because that was current in February when I built the machine, and I have not yet updated.

The disks are in two single-mirror pools. I plan to extend the non-root pool by one or two other mirrors once the disks get fuller, although I am not sure if I should rather keep one or two spare disks instead for safety. At the moment I am limited by the 4 SATA ports of the on-board host adapter, as the single PCIe slot is occupied by a network card for the DMZ.

Once I have move the networks (DMZ and internal) to VLANs on a single wire at the on-board interface, I can remove the network card and put in another SATA adapter. But there is no rush, as the disks are far from getting full soon. After all, disk space is now nearly 10 times as large as with the previous machine.

As the previous one, the server runs a lot of services for my wife and me and the neighbour family: DHCP, DNS (authoritative for my domains plus local resolver), SSH, FTP, Web server, HTTP Proxy, Mail with MTA and IMAPs including a few small mailing lists, Webmail interface, News, SMB (for the PVR), NFS (for my laptop), NTP. Even with all this, the machine is only very lightly loaded, so CPU power, with dual core @ 1.6 GHz and hyperthreading, is plenty. It was by far not enough for Dedup, though, which I had enabled for a short while — dramatically slowed disk I/O with long stretches of 100% CPU usage showed me the limit. Without Dedup, performance is very fine. ARC size is a little over 1 GB.

I do not use OpenSolaris’s auto-snapshot service, because with it different sets for each interval it makes unneccesarily many snapshots. I have written my own script to maintain only one set of regular snapshots, which are then thinned out as they get older. This is configurable per file system; for instance for the user file systems I make them every 10 minutes, which I keep for two hours, with a distance of an hour for a day, with a distance of 6 hours for 3 days, daily for two weeks, weekly for six weeks; on other filesystems with bigger distances as apropriate.

The previous server was a big success for five years, and I guess the new one will be one, too. Other than the previous one, the hardware platform is more flexible — I think I will add more disks as necessary, and perhaps replace the motherboard some time to get more RAM and more CPU power.

Going with OpenSolaris and ZFS was a very good move. I feel much more confident with all data being on weekly scrubbed mirrors, and I really cherish regular snapshots.

What do you mean by that? Zpool scrub killed your drive(s)?

This is all on Supermicro SAS chassis and hardware.
I’ve built quite a few of similar size to this, with the same disks, and had better results with OpenSolaris than Windows.

Although OpenSolaris is fussy about disks, and performance isn’t as good, I still prefer it for ZFS over the BSD version.
After zfs, linux file systems are simply unbearable. Try a chmod or rm -rf on 40Tb of files !!

Mark.

As of now (2010-05-27), I’m putting my trust in Samsung HD203WI hard drives, which is a 2TB drive.

In the consumer 2TB sector right now, this drive leads the pack in that (1) it has consistently high newegg.com user feedback, (2) apparently high reliability, (3) one of the lowest prices, and (4) seems to run cool.

This drive is a 3.5″ 4-platter, 500GB per platter drive, and spins at 5400 RPM instead of the usual 7200 RPM rotation speed. It has a min transfer speed of around 50MB/s on inner tracks, max speed of around 110MB/s on outer tracks, and an average transfer speed of around 80MB/s. In summary, for a SOHO NAS, this drive runs cool, quiet, is cheap, provides high capacity, and reasonable all-round performance, but is not the fastest drive on the block. Its performance is good, despite its low rotational speed due to high areal density. Also, it is possible to alter (reduce) the error-reporting time, which is interesting for RAID users to prevent the possibility of drives being kicked from the array, although this needs to be done at each boot — i.e. changes are not persistent.

I bought some of these drives, after reviewing all the available alternatives. You will find that other manufacturers of consumer 2TB drives are struggling to match the reliability record of this drive.

Cheers,
Simon

I was wondering if there have been any updates in the hard drive market since your January update.

All the best.

Thanks for blogging about your experiences all around.

My biggest concern is the RealTek 8112L NIC on the motherboard I’ve chosen, the actual ASUS board I’m using is on Sun.com’s HCL list for Solaris, so that’s good news. But we know how terrible RealTek chipsets CAN be.

That being said, I find it funny how many users running OpenSolaris look to that AOC-USAS-L8i card. I guess just because it’s pretty well priced. I’m eyeing it for future upgrades

Lsi/sun hba’s 20 on ebay. RE2,3 or NS drives $60 or less. Sas multiplier $300
Anything less than 12 drivves isn’t a NAS imo

I’m using an ASUS M4N78 Pro, and in addition to DRAM ECC enable, there are:
DRAM SCRUB REDIRECT
4-Bit ECC mode (chipkill – I read somewhere that AMD recommended enabling if using 4 DIMMS)
DRAM BG SCRUB
Data Cache BG SCRUB/L2/L3 Cache BG SCRUB

Do you have these options on yours as well?

Thanks

PS I just took off 500G for calculating the difference between Advertised and actual space, so don’t correct me.. Its just an estimate.

I expect WOL should work with an Intel GbE NIC, but I never tried it.

You can see quite high CPU utilisation when doing heavy transfers with the processor I have (Athlon X2 BE-2350), presumably the combined network and RAID checksum/striping/parity calculations, but it’s not been a problem, as the NAS spends most of its life idle or doing moderate reads and transfers.

Good luck and let us know how you get on with your choice of components. One thing, check out ECC memory.

Cheers,
Simon

I’ve found your blog very helpful and I’m now looking to take the plunge and build myself a NAS.

One aspect I’m most interested in is the power management side. I wondered if you or anyone had got any further with the WOL feature. I understand with your current NIC you found it didn’t work, but maybe people have had more luck with other cards?

Also, I’m thinking of holding off for the new Intel i3 processors. They should take advantage of the power saving features in Nahalem and have a lower max power usage. However, from your experience, would ZFS benefit from a more powerful processor?

David.

Whilst single bit error detection and correction in existing ECC is great to have, it seems highly desirable to have multi-bit error detection and correction as standard equipment for computer systems, and so perhaps there will be increased interest in memory correction technologies like IBM’s ‘chipkill’ which works by ’scattering the bits of an ECC word across multiple memory chips’ — see: http://en.wikipedia.org/wiki/Chipkill

Chipkill sounds like RAID for memory, where data is striped across the storage devices, memory chips in this case instead of hard drives. Sounds like it makes a lot of sense.

Quoting from that wikipedia page, there are other alternatives to IBM’s Chipkill tech from other hardware vendors:
“The equivalent system from Sun Microsystems is called Extended ECC. The equivalent system from HP is called Chipspare. A similar system from Intel is called SDDC.”

This REM (REliable Memory) from Divo Systems looks interesting. From their page at
http://www.divo.com/page4.html they say:

• Superior protection. For example REM-V6 corrects up to 12 bits in a 36-bit word! Also it allows up to four memory chips to fail completely without any affect on the SIMM’s behavior.
• Protection logic resides on the SIMM itself that makes it compatible with the regular PC memory and can be used in any existing memory slot. No custom adapters, special motherboards or time consuming installations required. It is just plugged in as any regular memory SIMM.
• Different levels of protection and various custom error correction schemes are possible.

Maybe Divo were too far ahead of the pack with this technology at the time? And the price was perhaps too high? And awareness in memory errors too low?

I saw the stuff on BackBlaze, but from what I remember, it looks like it will suffer from overloading, poor component choice and file system. If they had used LSI SAS/SATA controllers, Solaris and ZFS instead of the Linux/JFS combo, I think it would have been much more interesting and provide more solid data integrity and protection. And they didn’t use ECC memory???

Anyway, forget my quick critique on it, read a report from someone who has spent more time looking at the Backblaze design:
Some perspective to this DIY storage server mentioned at Storagemojo

But I think they are doing something important — showing that large amounts of cheap storage can now be a reality, just perhaps not using their current choice of hardware/OS/file system.

Cheers,
Simon

I am VERY interested in your home-nas-zfs solutions and even more so in your success using iScsi with MacBookPro’s

using 2009-06 opensolaris, and 10.6.1 snowleopard, and globalSAN 3.3.0.43 initiator — I did get it to work, and the persistant options works through a reboot — but — when I close the lid on the MacBook/Pro it hangs the sleep and I have to force-shutdown and reboot

Any ideas?

I’m planning on using the iScsi only to maintain a recent full backup of the laptops, via wireless of course.

thanks!

Al;

this report/blog has been _extremely_ interesting!!

p.s. — can one ONLY server a ZFS volume via iScsi? Should I be worried that the backing store is one great honkin’ file? Can I really trust in ZFS to protect all those bits in one file? (Or is this just my old-school supersitions?

There is a very interesting study on DRAM Error rates at http://blogs.zdnet.com/storage/?p=638
which comments on a Google study. DRAM error rates: Nightmare on DIMM street

It looks like ECC lover will be indicated for computers where important work is worked upon or stored.

on a related subject, have you seen the storagepod at http://storagemojo.com/2009/09/01/cloud-storage-for-100-a-terabyte/ and https://www.backblaze.com/petabytes-on-a-budget-how-to-build-cheap-cloud-storage.html

inetrsting isn’t it ?

best

Nicolas

Good luck with the mirror — it’s a really good idea to use a mirror for OS boot.
It will be easier if you use the whole drive for use within a mirror, as partial drive usage is not recommended within the ZFS Best Practices due to admin complexity etc, but maybe it will work well for you…

Thanks for the info on the Intel EX38 chipset, as I was not aware of that chipset. I originally did my hardware research around December 2007 for the current ZFS NAS I use, but when I did a quick search for EX38 and related motherboards I see dates from around mid-February 2008 onwards, so maybe this chipset & related motherboards became available a little after the time I researched?

At that time, it was the case with Solaris that if you wanted low power support using CPU frequency scaling for a 24/7 NAS then it was better to go for Intel processors, but for these Intel processors ECC support was hard to find within cheap commodity hardware (read: non-server stuff, except EX38 perhaps). Whereas with AMD processors you got ECC support with almost every cheap processor they made, but these cheap AMD processors generally didn’t support CPU frequency scaling, so for 24/7 NAS use, power costs became an issue. Things have moved on now though, and I look forward to building a new NAS with the more modern hardware available soon… However, turning the NAS off when not used is a simple solution.

Yes, PCI-X seems old tech now, and PCIe gets much better speeds and is also available on almost every motherboard.

Cheers,
Simon

Indeed, I have unearthed another 120G IDE drive which is likely to end as the boot mirror.
My system is double boot. I’ll guess I will use the space on the second drive which correspond to the XP partition on the first drive to be FAT-32 so I can nontheless easily ransfer data between OpenSolaris and XP.

The point about Intel chipset and ECC was about another of your post “http://breden.org.uk/2008/03/02/home-fileserver-zfs-hardware/”
in which you wrote:

#
Simon on January 7th, 2009 at 22:38
… Also, ECC memory support was important to me, and I don’t recall finding any suitable socket 775 compatible motherboards that supported ECC — but don’t quote me on that, as it’s a while back now…

But Intel have got some nice processors that support CPU frequency scaling for lower idle power consumption
…

In your blog, as in many discussing cheap systems supporting ECC, the Intel chipset EX38 is often forgotten.

but is is true that only a few manufacturer made motherboards properly for it.

Before settling on the Gigabyte EX38-DS4.
I tried the Asus P5 WS, which claimed to be server-class with a pci-x slot.
This slot does not run at more than 100Mhz.
Turthermore, the bios is badly programmed. It does not recognize expansion boards which have a too large ROM, thus preventing using some Hard disk adapters.

Nicolas