Lsi/sun hba’s 20 on ebay. RE2,3 or NS drives $60 or less. Sas multiplier $300
Anything less than 12 drivves isn’t a NAS imo

I’m using an ASUS M4N78 Pro, and in addition to DRAM ECC enable, there are:
DRAM SCRUB REDIRECT
4-Bit ECC mode (chipkill – I read somewhere that AMD recommended enabling if using 4 DIMMS)
DRAM BG SCRUB
Data Cache BG SCRUB/L2/L3 Cache BG SCRUB

Do you have these options on yours as well?

Thanks

PS I just took off 500G for calculating the difference between Advertised and actual space, so don’t correct me.. Its just an estimate.

I expect WOL should work with an Intel GbE NIC, but I never tried it.

You can see quite high CPU utilisation when doing heavy transfers with the processor I have (Athlon X2 BE-2350), presumably the combined network and RAID checksum/striping/parity calculations, but it’s not been a problem, as the NAS spends most of its life idle or doing moderate reads and transfers.

Good luck and let us know how you get on with your choice of components. One thing, check out ECC memory.

Cheers,
Simon

I’ve found your blog very helpful and I’m now looking to take the plunge and build myself a NAS.

One aspect I’m most interested in is the power management side. I wondered if you or anyone had got any further with the WOL feature. I understand with your current NIC you found it didn’t work, but maybe people have had more luck with other cards?

Also, I’m thinking of holding off for the new Intel i3 processors. They should take advantage of the power saving features in Nahalem and have a lower max power usage. However, from your experience, would ZFS benefit from a more powerful processor?

David.

Whilst single bit error detection and correction in existing ECC is great to have, it seems highly desirable to have multi-bit error detection and correction as standard equipment for computer systems, and so perhaps there will be increased interest in memory correction technologies like IBM’s ‘chipkill’ which works by ’scattering the bits of an ECC word across multiple memory chips’ — see: http://en.wikipedia.org/wiki/Chipkill

Chipkill sounds like RAID for memory, where data is striped across the storage devices, memory chips in this case instead of hard drives. Sounds like it makes a lot of sense.

Quoting from that wikipedia page, there are other alternatives to IBM’s Chipkill tech from other hardware vendors:
“The equivalent system from Sun Microsystems is called Extended ECC. The equivalent system from HP is called Chipspare. A similar system from Intel is called SDDC.”

This REM (REliable Memory) from Divo Systems looks interesting. From their page at
http://www.divo.com/page4.html they say:

• Superior protection. For example REM-V6 corrects up to 12 bits in a 36-bit word! Also it allows up to four memory chips to fail completely without any affect on the SIMM’s behavior.
• Protection logic resides on the SIMM itself that makes it compatible with the regular PC memory and can be used in any existing memory slot. No custom adapters, special motherboards or time consuming installations required. It is just plugged in as any regular memory SIMM.
• Different levels of protection and various custom error correction schemes are possible.

Maybe Divo were too far ahead of the pack with this technology at the time? And the price was perhaps too high? And awareness in memory errors too low?

I saw the stuff on BackBlaze, but from what I remember, it looks like it will suffer from overloading, poor component choice and file system. If they had used LSI SAS/SATA controllers, Solaris and ZFS instead of the Linux/JFS combo, I think it would have been much more interesting and provide more solid data integrity and protection. And they didn’t use ECC memory???

Anyway, forget my quick critique on it, read a report from someone who has spent more time looking at the Backblaze design:
Some perspective to this DIY storage server mentioned at Storagemojo

But I think they are doing something important — showing that large amounts of cheap storage can now be a reality, just perhaps not using their current choice of hardware/OS/file system.

Cheers,
Simon

I am VERY interested in your home-nas-zfs solutions and even more so in your success using iScsi with MacBookPro’s

using 2009-06 opensolaris, and 10.6.1 snowleopard, and globalSAN 3.3.0.43 initiator — I did get it to work, and the persistant options works through a reboot — but — when I close the lid on the MacBook/Pro it hangs the sleep and I have to force-shutdown and reboot

Any ideas?

I’m planning on using the iScsi only to maintain a recent full backup of the laptops, via wireless of course.

thanks!

Al;

this report/blog has been _extremely_ interesting!!

p.s. — can one ONLY server a ZFS volume via iScsi? Should I be worried that the backing store is one great honkin’ file? Can I really trust in ZFS to protect all those bits in one file? (Or is this just my old-school supersitions?

There is a very interesting study on DRAM Error rates at http://blogs.zdnet.com/storage/?p=638
which comments on a Google study. DRAM error rates: Nightmare on DIMM street

It looks like ECC lover will be indicated for computers where important work is worked upon or stored.

on a related subject, have you seen the storagepod at http://storagemojo.com/2009/09/01/cloud-storage-for-100-a-terabyte/ and https://www.backblaze.com/petabytes-on-a-budget-how-to-build-cheap-cloud-storage.html

inetrsting isn’t it ?

best

Nicolas

Good luck with the mirror — it’s a really good idea to use a mirror for OS boot.
It will be easier if you use the whole drive for use within a mirror, as partial drive usage is not recommended within the ZFS Best Practices due to admin complexity etc, but maybe it will work well for you…

Thanks for the info on the Intel EX38 chipset, as I was not aware of that chipset. I originally did my hardware research around December 2007 for the current ZFS NAS I use, but when I did a quick search for EX38 and related motherboards I see dates from around mid-February 2008 onwards, so maybe this chipset & related motherboards became available a little after the time I researched?

At that time, it was the case with Solaris that if you wanted low power support using CPU frequency scaling for a 24/7 NAS then it was better to go for Intel processors, but for these Intel processors ECC support was hard to find within cheap commodity hardware (read: non-server stuff, except EX38 perhaps). Whereas with AMD processors you got ECC support with almost every cheap processor they made, but these cheap AMD processors generally didn’t support CPU frequency scaling, so for 24/7 NAS use, power costs became an issue. Things have moved on now though, and I look forward to building a new NAS with the more modern hardware available soon… However, turning the NAS off when not used is a simple solution.

Yes, PCI-X seems old tech now, and PCIe gets much better speeds and is also available on almost every motherboard.

Cheers,
Simon

Indeed, I have unearthed another 120G IDE drive which is likely to end as the boot mirror.
My system is double boot. I’ll guess I will use the space on the second drive which correspond to the XP partition on the first drive to be FAT-32 so I can nontheless easily ransfer data between OpenSolaris and XP.

The point about Intel chipset and ECC was about another of your post “http://breden.org.uk/2008/03/02/home-fileserver-zfs-hardware/”
in which you wrote:

#
Simon on January 7th, 2009 at 22:38
… Also, ECC memory support was important to me, and I don’t recall finding any suitable socket 775 compatible motherboards that supported ECC — but don’t quote me on that, as it’s a while back now…

But Intel have got some nice processors that support CPU frequency scaling for lower idle power consumption
…

In your blog, as in many discussing cheap systems supporting ECC, the Intel chipset EX38 is often forgotten.

but is is true that only a few manufacturer made motherboards properly for it.

Before settling on the Gigabyte EX38-DS4.
I tried the Asus P5 WS, which claimed to be server-class with a pci-x slot.
This slot does not run at more than 100Mhz.
Turthermore, the bios is badly programmed. It does not recognize expansion boards which have a too large ROM, thus preventing using some Hard disk adapters.

Nicolas

Which ’slight error regarding ECC and Intel chipsets’ did you mean?

Simple 2-drive mirror NAS devices, such as you mention, might appeal to some people, especially on price, ease of use, and power requirements, but these devices lack automatic detection & repair of latent errors (bit rot) when a file is read back. And they have no scrub capability which can run on a regular basis.

I agree that the Boot Environment functionality in OpenSolaris is a great feature, and I also use it, and it has saved my system when it failed to boot after performing an ‘update all’ to update all packages. Like you say, after getting a boot failure situation, one simply selects the previous boot environment from the GRUB menu. Simple and effective!

ZFS admin, although reasonably simple, is a learning curve and most people are not technically-minded: they just want something that plugs in and works. One day, when one of the existing NAS manufacturers realises the enormous appliance potential for OpenSolaris, then hopefully the general public can also benefit from robust data storage protection without needing sys admin skills. Maybe Oracle will delve into this market?

In your ‘next steps’ under item H, you mention about protecting the boot disk from failure. You might also wish to consider using a mirror for your boot pool to make a more robust boot environment. Then you have the dual protective features of (1) a mirror and (2) Boot Environments combined. If that sounds interesting, see here:
Home Fileserver: Mirrored SSD ZFS root boot

I found your blog articles to be a very good source of information. The concrete examples pushed me over the edge.

Yet I wanted to correct a slight error regarding ECC and Intel chipsets

I am witing this on my OpenSolaris 2009-06 box
which is using an INTEL ECC Motherboard : Gigabyte EX38-DS4 (this Express 38 chipset can be used with either ECC and non ECC) with 6 SATA-2 on board, 2 PCI-Express and 2 GigE network port
with 4 Gig of ECC RAM
CPU CoreDuo E2180
6* Samsung F1
1* Maxtor 120 (dual boot XP and Solaris)
2*DVD-Writer
Corsair 650W PSU (a tad overboard, but I may upgrade the videocard
Nvidia Quadro NVS-280 PCI card
1 DELL SATA-SAS card to get 4 additional SATA in one PCI-Express slot (could add another 8-port SATA-SAS in the other)
in an Antec 300 box which is one of the nicest box ever used and extremely well ventilated (with 2 Noctua 12cm fans on top of the 2 14cm fan factory mounted)

The EX38-DS4 is said to be obsolete despite its advanced features, I got it for cheap. Most of my hardware is Intel/CoreDuo Intel/Quad, so no AMD this time. I am unaware of a MB with EX38 and integrated video.

This is mostly commodity Hardware. The plan call for a second machine build from left-over component (perhaps even cheaper with normal RAM and Z1) to be synchronized at 300km

I also want to offer to the poor souls battling with X11 on OpenSolaris the following:

If I was to sum up my experience of OpenSolaris:
A. Getting a copy on Cd-Rom was tough, but creating a bootable USB-Stick was simple enough and works even better (thanks to http://www.genunix.org)
B. Connecting to Internet was easy (clicking on the appropriate icon…)
C. Updating the packages was trivial using the “add more software” button
D. Transferring files from XP computers on the network was painless using the File browser Nautilus to browse the network
E. Setting up the HP Lserjet networked printer required a dive into the book “OpenSolaris Bible” by Solter, Jelinek and Miner to install and enable CUPS
BUT
F. Getting the display to run 1920*1080 in Gnome was a hair-tearing ordeal.

===== beadm plug ======
Before I lost my sanity (and after 2 full reinstall), I managed to understand how the BootEnvironment work, and what a difference it makes. The OS utility beadm is a godsend!

beadm allowed me to save full bootable setups more and more refined.
When a Nvidia driver too recent for my hardware or an incomplete xorg.conf file prevented the system from completing the boot, I used grub to boot the previous working environment and then mounted the non-functionning boot environment to discover the errors stored in the X log file, then modified the xorg.file in the test environment before unmounting and attempting another boot.
===== end of beadm plug =====

So, readers with Nvidia NVS-280 may follow the steps I had to take:
1. remove the nvidia graphics package
2. install the 173.14.20 drivers
3. generate a default xorg.conf with nvidia-xconfig
4. modify the xorg.conf file to match the monitors resolution (discovered using Powerstrip under Microsoft XP)
# LG 2361V-FP
DisplaySize 510 287
HorizSync 30.0 – 83.0
VertRefresh 56.0 – 75.0
ModeLine “1920×1080″ 173.605 1920 2056 2264 2576 1080 1081 1084 1118 -hsync +vsync
ModeLine “1600×1024″ 136.617 1600 1720 1888 2144 1024 1025 1028 1060 -hsync +vsync
ModeLine “1280×1024″ 107.981 1280 1344 1456 1688 1024 1025 1028 1066 +hsync +vsync
ModeLine “1024×768″ 65.027 1024 1064 1200 1344 768 771 777 806 -hsync -vsync
ModeLine “800×600″ 39.971 800 856 984 1056 600 601 605 628 +hsync +vsync
5. rebot
6. use the System/Preferences/Screen Resolution application to change settings.

Next steps:
G. Finalize the Raid-Z2 setup
H. Use beadm to create a copy of the stable BootEnvironment on the ZFS storage (to protect from a failure of the boot disk)
I. Connect the UPS
J. Consolidate all pictures and music onto the system
K. Create ISO from DVD
L. Serve media to the household: Pics, Music, and films
M. Upgrade the router to a trunking-capablen router
N. Set up remote server
O. Automate sync between servers

In the end, a ZFS Home server is not for the faint of the heart. Unless you have an OS background and you NEED large capacity with multiple HD and you are UTTERLY convinced by ZFS, a commercially available NAS such as a QNAP TS-219 Raid-1 NAS with 2*1To is very easy to setup, and eats VERY VERY VERY little power.

I have been using OpenSolaris and ZFS for 18 months now. I found ACL descriptions in the SUN documentation totally confusing. I have been using the following commands to get my ACL’s under control. The following commands have been working for me. The last 6 lines of chmod are the key to success. After I executed the last 6 lines, all my ACL problems were solved

I am no expert, so please don’t just copy paste these commands. Understand them first…….

I have 2 ZFS filesystems, one “/tankbig/tank”. This is read only for everyone, but only the file owner can write. The second one, “/tankbig/share” everyone can read and write.

chmod -R 750 /tankbig/tank
chmod -R 777 /tankbig/share

chown -R brian:shareme /tankbig/tank
chown -R brian:shareme /tankbig/share

chmod -R A=owner@:full_set:file_inherit/dir_inherit:allow /tankbig/tank
chmod -R A+group@:read_set/execute:file_inherit/dir_inherit:allow /tankbig/tank
chmod -R A+everyone@:read_set/execute:file_inherit/dir_inherit:allow /tankbig/tank

chmod -R A=owner@:full_set:file_inherit/dir_inherit:allow /tankbig/share
chmod -R A+group@:full_set:file_inherit/dir_inherit:allow /tankbig/share
chmod -R A+everyone@:full_set:file_inherit/dir_inherit:allow /tankbig/share

Brian.

Glad you enjoyed building your own ZFS-based NAS and mostly got it working nicely now.

You’re right, I didn’t use Windows clients much to access the ZFS-based NAS so I haven’t encountered and found solutions to Windows-related access/permissions issues. Maybe later

So far I’ve mostly accessed the NAS from a Mac.

The problem you mention of the slow-loading game is curious. My first question was going to be to ask if you had (1) Gigabit NICs on both computers, (2) Gigabit switch and (3) Category 5e or Category 6 ethernet cabling, but as you mention speeds of 50+ MBytes/sec speeds then that seems OK.

So the issue looks like something to do with the latency differences between loading directly off local hard drive and the NAS across the network. A pure guess would be that the game load process involves the loading of many little files and that this could cause a small speed difference for each file load to be magnified. But it’s a pure guess.

You could perhaps fire up the network diagnostics on both ends of the network next time you load the game and see what speeds you see on each end sending and receiving.

Then compare those speeds with the disk activity diagnostics when loading the game directly from local hard drive. Doing this might yield some clues. Let me know what you discover.

Have fun!

Cheers,
Simon

thanks to your very enjoyable posts (from a geek’s point of view), I have been the owner of a homemade ZFS-based NAS for about two weeks now. It’s working great so far, even though I struggled a bit when authenticating from Windows to the CIFS server at first. This is now resolved, but it would have been nice if you had developped that part of the installation process in one of your articles (unless you never encountered any such problem, in which case you don’t know your luck). Comparatively, accessing the same shares as an NFSv4 filesystem from Linux was painless, even setting up the UID/GID mappings went smoothly. I haven’t tried from a Mac yet, but it should be even more painless.

Anyway, I actually wanted to warn about a small usage issue that I have bumped into during my tests, and maybe someone could help me understand it, and maybe even fix it.

I’ve always planned this NAS to mainly hold three kinds of file: my photos (those 14 megapixels RAW shots sure take some place), my music & movies, and finally games. As hinted earlier, my main computer is a Windows/Linux dual-boot setup, and I use the Microsoft OS to play games. That’s roughly it. The issue that has me puzzled is that, even though I’ve bought a small D-Link Gigabit switch serving its purpose perfectly when playing music or movies, moving files around, or when browsing & managing my photos, launching applications from the NAS has proved to be quite a problem. It works, mind you, but with a very annoying symptom: a game that used to launch in about 10 seconds now takes almost 1 minute to be available. These launch times were measured in a simple manner: start a stopwatch as I double-click the executable, stop it as I reach the main menu. After that, loading times in the game itself are very much the same as what I used to experience when it was installed on my internal 320GB hard drive. I have looked around for clues, but the info I gathered was either outdated (when Gigabit home networks where neither practical nor possible), or irrelevant to my situation (i.e. an application server that actually *runs* the applications and streams them to clients).

Alone and facing these cold machines, I poked away at a few settings, without any of them making any difference: compression=on/off, atime=off, deactivating Windows security warnings (i.e. running an executable from a network drive), etc. There must be some technical aspect to loading-an-executable-into-memory-and-then-running-it that I am not aware of, because I don’t see why it would take that much longer when done off of a network drive (again, especially when only the *loading* part is actually crippled, the running part is *fine*).

But overall, this NAS delivers: a few different tools gave me between 50 and 75 MB throughput, managing zpools and zfs datasets is amazingly easy and satisfying, though I’ve yet to fully tame those beasts, and the price is much lower than any comparable commercially avaiable product. It could definitely become the be-all-end-all when (and if) I solve that annoying loading times issue.

One interesting benchmark came from the use of the Intel NAS Performance toolkit available at (warning, this is a WinXP-32-bits-on-Intel-processor-*ONLY* tool). You can test different workloads, and as the best one gave me 77MB/s, some where deep down in the 5MB/s area. From what I understood, such a difference can be blamed on sequential vs. random file access. The whitepaper was also interesting, and one part of it actually makes me think that yes, running applications from a NAS is a different beast altogether: “Consistent with mainstream personal computer usage, we assume that operating system files and executables will be kept locally on the client. (…) Therefore, our traces include only those transactions targeting data/media files and initiated by the application being traced. System generated accesses and accesses to executable program files are excluded.”

So once again, thanks for all your helpful posts on the matter, and see you in a year when I take a look back on my experience.

Laurent.

I agree that one would save time adding 2 drives once, rather than 1 drive twice, and you’re right — having more space available than we need right now is a good idea. I just used similar sized disks for consistency to ensure it worked well, rather than upgrading to bigger drives — but that is a good option I will use on another system sometime.

Speed is a little irrelevant here right now as I’m currently using a single GbE connection limiting network transfer speeds to around 50 MBytes/sec, so having 2 RAID-Z1 vdevs is not of interest to me with this NAS, but it might be of interest to others with different configurations, or who often need to transfer large amounts of data within the NAS — i.e. not across the network.

Failing drives are not fun — good luck with your backup!

Cheers,
Simon

For the resilvering, I agree it is a strain on your drives and if you are really worried about it, but then again what about normal use and if the disks are that bad maybe you should have changed them b4.
You can lower the change of it going wrong while resilvering with a disk you did not exchange. By doing a scrub beforehand and, although this one is tuff on solaris, an extended smart check.
Now compare the disks for their smart data and choose the one you think is going to fail the most likely. Personally I think that after the previous to intensive operations the disk will survive the resilvering process just fine. And you can check before changing the next disk the smart data and choose again accordingly.

And for safety of the 2 raidz vdevs versus 1 raidz2 I know you do not get extra safety, I just mentioned that instead of safety you get a bit more speed, since the 2 raidz will be “striped” by zfs.

p.s. my experience with failing harddrives is not very pleasant either. In fact right now I am making an one to one backup of my laptop drive since it had some pauses…. just a gut feeling but the previous 2 gut feelings were correct so better be safe the sorry here.

Like you said, if you’re on a tight budget then adding a single drive to an existing vdev, like a raidz vdev, if it were possible right now, would be an interesting possibility.

But like you say, other ways of upgrading the storage pool do exist. I think in previous posts, some people were confused when I said you can’t expand an existing raidz vdev by adding additional drives, and thought I was saying that you can’t expand a zpool storage pool, which is not true, of course, as you correctly pointed out.

For me personally, the second option you mentioned of adding a new raidz vdev was not an option due to lack of SATA ports on the motherboard. And the first option of swapping a disk at a time frightens me a lot, due to the multiple resilvering process required. See these comments:
http://breden.org.uk/2008/09/01/home-fileserver-raidz-expansion/#comment-4046
http://breden.org.uk/2008/09/01/home-fileserver-raidz-expansion/#comment-4057

And of course, comparing the two configurations of (1) two 3-drive raidz1 vdevs, against (2) one 6-drive raidz2 vdev, we can see that both (1) and (2) use the capacity of 4 drives for data and the capacity of 2 drives for parity, but the important difference is that (2) has double parity across the whole raidz2 vdev, and so is more protected against data loss — i.e. (2) can survive any 2 drives failing before data loss occurs, whilst (1) can only survive one drive failing in each of the raidz1 vdevs — if two drives fail in either raidz1 vdev then data loss occurs.

You’ve probably already discovered all you need to know by now regarding ACLs to be used in Windows environments, but if not there is a useful post I found that was posted by an experienced Windows systems administrator that might be interesting — see here:
http://breden.org.uk/2009/05/10/home-fileserver-zfs-file-systems/#comment-9524

I had mixed results with the trunking too, but it was definitely a lot faster when I had it working. But the trunk suffered from a buggy 2nd GbE port that often failed to initialize on the Asus M2N-SLI Deluxe motherboard.

Like yourself, I too have run out of SATA ports: soon it’s time to get another SATA card too

Again, like yourself I’m looking to mirror the OS boot disks, and due to breaking a pin on the boot drive’s IDE connector, I took the opportunity to install OpenSolaris 2009.06. However, as I now want to use this NAS to do more stuff, my configuration is likely to get more complex and so I’m now thinking of creating a mirrored, multiple boot environment-capable setup using OpenSolaris 2009.06 with a couple of SSDs, as this case has run out of vibration-dampened, silicone-grommeted drive bays, so it’s SSDs for me, despite their ridiculous $/GB cost right now, but two 30GB SSDs should do nicely.

Thanks for your posts regarding ZFS. They were very helpful. I built a server last year with the following hardware:

CPU: AMD X2 4850E
MB: XFX MDA72P7509 NF750A (6x SATA, 2x PCIe x16 ports)
RAM: 4GB ECC RAM

The onboard NIC is a Marvell 88E8056 and is not supported, as of OpenSolaris 2009.06. I used an Intel 1000GT PCI NIC instead. Initially I had two of these Intel NICs and set them up as a trunk on a Cisco 2970, but I didn’t get the expected results. The 802.3ad standard does not round robin packets like you’d think it would. It uses src/dest hashing instead. Also, the eSATA port on the motherboard requires a SATA pass through cable to be plugged into one of the six ports on the motherboard. So if you want to use eSATA, you can only use 5 internal SATA ports. I may get a cheap pci-express eSATA card later.

I was using a single 80 GB IDE drive for the OS, and 6x 300GB SATA drives in raidz. Those drives are old and way past their warranty, so I decided to upgrade the drives with 1TB drives. I only bought 2 so far at $90 each. I’ll buy more later.

My new storage is configured like this:

OS: 2x 300GB SATA drives in a mirror (old, but never used, were spares)
DATA: 2x 1TB SATA drives in a mirror (will soon expand to a 4x 1TB raidz2)

That will max out my SATA ports. I decided to mirror the OS since it should increase performance and redundancy. I am also no longer running SXCE. It was too much work and I don’t like SVR4 style package management and I don’t like bi-weekly upgrades. I prefer to use IPS, so I am running OpenSolaris 2009.06.

Thanks,
Simon