Yep, it’s easy to get really low power stuff, but it doesn’t have ECC. For example look at the Intel Pine Trail platform, including the new Pineview processors which are single/dual-core 1.66GHz 64-bit second generation Atom processors, which integrate processor, chipset, memory controller and GPU, and use incredibly low amounts of power ranging from 5.5W TDP for the single-core Intel Atom N450, through the top of the range dual-core Intel Atom D510 which is a miserly 13W TDP. Unfortunately for ZFS-usage, they don’t support ECC memory, which makes them unsuitable where data integrity is of paramount importance.

Also see my comments on the Athlon II X2 ‘e’ range of processors here: Home Fileserver: A Year in ZFS

Like you say, once you start venturing into server territory then the prices get jacked up considerably. But the server components path is easier to justify cost-wise if the NAS will be used for a business.

Indeed, it is wise to consider using sufficient redundancy, and I consider RAID-Z2 a lot more robust than RAID-Z1. Some people prefer to use 3-drive mirrors to attain the equivalent protection of double parity you get with RAID-Z2. For example, see Constantin Gonzalez’s post here: Home Server: RAID-GREED and Why Mirroring is Still Best.

Mirrors do offer big advantages over RAID-Z1/2/3 as each drive contains the whole data. In the author’s case in the article above, his data is limited to 1.5TB, as he has two 1.5TB drives and one 2TB drive within the 3-drive mirror. To get more than 1.5TB he will have to upgrade the other two 1.5TB drives to 2TB drives, giving 2TB in total. Or to break above the 1.5TB or 2TB level, he will have to buy another three drives, assuming he sticks with 3-drive mirrors. So for 3 drives he can currently get 1.5TB or 2TB on upgrading, or for 6 drives he could have 4TB of usable data space, using all 2TB drives. That system would give 4TB usable data space, and 8TB in additional mirrored data across 6 drives.

The equivalent 6-drive RAID-Z2 vdev would have 4 data drives of 2TB and 2 drives for parity. So 8TB usable data space, 4TB for parity. Both methods have different advantages. Of course, RAID-Z3 is another option and gives even more protection.

For booting, you will probably have to use a mirror vdev. For your data pool, either use a mirror like Constantin above, or use RAID-Z2 / RAID-Z3. RAID-Z1 is considered too little redundancy these days with large data pools formed with modern high-capacity SATA drives.

As mentioned in my comment on the hardware page, ZFS will heal both mirror and RAID-Z1/Z2/Z3 vdevs when errors are detected on files being read or a scrub is performed.

Also on power failure, ZFS always leaves the file system in a consistent state, as it is transactional. Either the data was written, or it wasn’t, but you won’t get a half-written inconsistent file system state even if power is lost during a write.

Good luck.

Cheers,
Simon

Thanks for the “Rome wasn’t build in a day” expression, I will have to use that more often. Maybe frame it and put it on the wall for ready reference.

You and I share the same philosophy re: the do it right. When it comes to computer stuff, the “project triangle” of good, fast, cheap starts to come unstuck these days. If you want good, there is no substitute for doing the research. Paying money doesn’t guarantee getting a solution either. Which means that if you want good, you just have to come to grips with the fact that it will take time. But on the plus side, the solution is often cheaper than you think.

As far as the server components go, they certainly don’t make it easy to get low power consumption for a low price. I seached for “opteron HE” on ebay, and found one for $100, but the mobo would be $300 or more I would say. But still, when you compare to the Thecus N7700 which is billed as being for the “enterprise” and doesn’t run RDIMMs, you can really build a premium quality NAS yourself for less money.

Probably more cost effective to just go RAIDZ+2 rather than worry about the quality of the drive (e.g. RAID drives).

In the end I made a decision by separating work and home needs. For work I need the ultimate in reliability. The workstation will have a Xeon (X3450), ECC RDIMM, ZFS. I’m going to have 3 * OCZ Vertex 30GB in RAIDZ as my boot (I think this is possible?). For more breathing room, I’ll have another RAIDZ (pool?) made up of a few regular SATA HDD, probably RAIDZ2. I think that should be able to survive anything bar a lightning strike etc. Which would probably take out both systems if I were to set up a server-grade work NAS. I suppose a NAS is more theft resistant though because you can hide it.

I’ll back the HDD RAIDZ up after significant changes to a HDD which I’ll then take offsite.

I had thought about combining work and home needs of storage in a NAS but then thought better of it (security mainly). So now what I’m going to do (because I’ve still got heaps of other media to serve and that sort of thing) is to make my family PC (which runs Ubuntu) do that duty. (Should do it well – it’s a 780g/ 4850e, doesn’t use much power and is almost silent thanks to the P182 + Scythe Ninja + ST30NF passive power supply). Not sure if I really want to make it run Solaris or FreeBSD though. I will back it up in a similar fashion to the work PC. If I get a single OCZ Vertex 30GB for a boot drive and run mirrored HDDs, that will probably be good enough in combination with the backup. As far as bit rot, if a photo flips a pixel it’s probably not that big a deal. I worry a bit about RAID 5 and power failure, I suspect that mirroring is more fault tolerant in that regard.

Anyway, thanks for your reply.

Personally, I believe that if you’re going to do something, you might as well do it right, even it means more work. After all, what good is an unsatisfactory non-solution?

This was an important project for me, as I had already lost a lot of photos, and just managed to recover them again, but only after a lot of work. But I could so easily not have been so lucky. From then on, I was on the look out for a good solution to the big storage problem.

So this was an interesting task, and as Henry Ford once said: ‘Interesting work is easy and I am always certain of results.’

Yes, I’m sure you’re right that using registered memory would provide even greater data safety properties than using standard un-buffered ECC memory, but you are right that there can be quite a substantial cost premium attached to upgrading the CPU to Xeon/Opteron, using registered memory and a suitable enterprise quality motherboard. Also RAID-edition drives too perhaps?

So far, I have tried to keep costs down by using mainstream, commodity, non-exotic hardware and things seem to have worked out OK. One day I might take the more expensive approach.

ZFS in FreeBSD 8.0 could well be good, but personally I like to use the ‘reference implementation’ of ZFS from Sun running on OpenSolaris, as it means that the latest bug fixes and features are available if required, even if one has to research hardware a little more carefully, which is not that difficult, with things like the Solaris HCL and zfs-discuss forum available.

I don’t think that Oracle will do anything silly with OpenSolaris, but I could be wrong of course. It’s useful to find problems before rolling out the robust enterprise version: Solaris. And Ellison loves Oracle on Solaris, so I expect things to remain as they are.

Thanks for the memory doc link — I will take a read.

Yeah, tell your wife that Rome wasn’t built in a day

Best of luck,
Simon

I’m just noticing your attention to everything – you are as anal in researching and optimizing as I am! I note the P182 – excellent case. I’ve had one for my main system for over a year now. It must have taken you man-weeks or man-months to do all the research for this (and things in general). I am about 3 weeks into researching my workstation and home network build, and my wife wonders why I haven’t got anything tangible yet to show for it. You can’t rush quality.

I will have to read more of your blog.

I have come along much the same path as you have it seems, before discovering your blog. Well done for writing it all out! One thing that you or other people might want to do if your data is absolutely mission critical – strongly consider using not just ECC memory, but registered (i.e. buffered) memory (i.e. RDIMM). This will increase reliability. After all, if there is an error that occurs due to the state of the RAM (which the ECC will not catch due to the lack of buffering), then the ECC won’t help you and the ZFS won’t help you either.

Of course, this does appear to necessitate (based on much googling) that you will be restricted to either an Opteron or a Xeon processor (I would love to be proven wrong of course! If some cheaper alternative exists please let me know.). I also believe that the server CPUs are from a more conservative bin relative to their clock/voltage. So you may have a bit more security there too if you go with a server CPU.

I’m sure the price discrepancy between DIY and commercial NAS really opens up when you take this approach – sure, you might spend $400 for mobo/CPU, but all the other parts and software will be identical. So for at most $300 more, you are getting a truly enterprise class NAS. (Well, you might need to throw a redundant power supply in there, but other than that…)

I’m also strongly considering going with FreeBSD. Version 8.0 has been released and they say that ZFS is no longer in experimental status. From what I know, FreeBSD has a better reputation than Linux for stability. I would therefore take this statement with less of a grain of salt than most other sources.

I’m also worried about the influence of Oracle on OpenSolaris. For my workstation and in future, server, I’m going to be running PostgreSQL. Oracle have an incentive to make things not work so well for PostgreSQL in future, so I think FreeBSD will be a safer option. But for a NAS, OpenSolaris can work fine I suppose.

http://images10.newegg.com/UploadFilesForNewegg/itemintelligence/NI_System-Memory/NIC-Pro-Do_I_Need_ECC_and_Registered_Memory-v1.1e.doc

Having no experience of Solaris and little of RAID makes you as qualified as I was when I started this project, so congratulations!

Regarding the possibility of non-enterprise drives being kicked from a RAID-Z1 or RAID-Z2 vdev due to longer error response times, I’m afraid I don’t know, but it sounds quite possible and likely.

However, I know Western Digital (WD) have a utility called WDTLER.EXE which you can run on each non-enterprise WD drive you add to your RAID array to make them report errors quickly in the same way as their enterprise models.

More info here:
http://en.wikipedia.org/wiki/Time-Limited_Error_Recovery

Having said that, I didn’t use that utility on my WD drives and they have not given even one read, write or checksum error in almost 2 years of regular storage pool scrubs since I built this machine. Rightly or wrongly, I have a hunch that this good fortune is due to a number of factors:

1. Good damping of drives within the case using silicone grommets. This greatly reduces drive mechanism vibrations from being transmitted to a shared case chassis and then back to the other drives in the array. If this sounds hard to believe, consider that shouting at drives has been shown to cause interesting effects — check out this fun video and story: http://blogs.sun.com/brendan/entry/unusual_disk_latency
2. Good choice of highly-rated, reliable, yet cheap commodity SATA drives.
3. Stable circa 40 degrees Celsius temperature operating environment across all drives (script reports temps, fans can be adjusted if required)
4. Case is off the floor away from footstep-induced vibrations (and dust).
5. I turn the NAS off when not in use, so the cheap non-enterprise drives have less operating hours, although turning the system on and off each day could be worse on the drives than leaving it on 24/7, so maybe this one is not true…???
6. An electrical power conditioner to smooth the electrical supply to the NAS, built into the power outlet.

Good luck with your new system!

Cheers,
Simon

First of all, thank you for putting up such a detailed log of your experiences with ZFS. It’s inspired me to start planning out my own ZFS home file server, even though I have no experience with Solaris and little with RAID.

I do have a question I hope you can answer. In regular RAID arrays, enterprise-level hard drives are recommended because they have a shorter timeout period. When bad sectors are encountered, they let the RAID array recover the data using parity information on other hard drives. On the other hand, consumer-level hard drives are not recommended because they try to recover bad sectors themselves. When the drives don’t respond quick enough, the RAID array marks the drive as bad and kicks it out of the array. Are you aware of whether this is something that will affect a RAIDz1 or RAIDz2 array?

Thanks!

Ap_Id Type Receptacle Occupant Condition
sata4/0 unknown connected unconfigured unknown
sata4/1 unknown connected unconfigured unknown

One thing i havent tried is installing windows on to that harddrive both connected to the JMicron,
but i think its unneccessary because the “Hiren’s BootCD 9.8″ booted fine.

what do you think?

First though, maybe see if you can find any users reporting success with the same mobo, because there may be some driver issue perhaps.

Also, there might be some tweaks within the BIOS that are required.

If you find out how to get it working, it would be interesting to hear from you, as I didn’t have much luck and gave up trying.

Cheers,
Simon

However, I think it should be possible to access a drive plugged into the JMicron SATA port, so maybe there is some problem, like you say.

I presume you enabled the JMicron controller in the motherboard’s BIOS settings?

Cheers,
Simon

im getting the error:

Jul 26 15:17:05 Videodrome genunix: [ID 936769 kern.info] fd0 is /pci@0,0/isa@1/fdc@1,3f0/fd@0,0
Jul 26 15:17:16 Videodrome ahci: [ID 860969 kern.warning] WARNING: ahci0: ahci_port_reset port 0 the device hardware has been initialized and the power-up diagnostics failed
Jul 26 15:17:28 Videodrome ahci: [ID 860969 kern.warning] WARNING: ahci0: ahci_port_reset port 1 the device hardware has been initialized and the power-up diagnostics failed

and as you can see it adds about 10 seconds to the boot time when its searching for the drives.

(the drives connected are at sata4/0 and 4/1)

#cfgadm -la
Ap_Id Type Receptacle Occupant Condition
sata1/0::dsk/c9t0d0 disk connected configured ok
sata1/1::dsk/c9t1d0 disk connected configured ok
sata2/0::dsk/c10t0d0 disk connected configured ok
sata2/1::dsk/c10t1d0 disk connected configured ok
sata3/0::dsk/c11t0d0 disk connected configured ok
sata3/1::dsk/c11t1d0 disk connected configured ok
sata4/0 unknown connected unconfigured unknown
sata4/1 unknown connected unconfigured unknown
usb5/1 unknown empty unconfigured ok
usb5/2 unknown empty unconfigured ok
usb5/3 unknown empty unconfigured ok
usb5/4 unknown empty unconfigured ok
usb5/5 unknown empty unconfigured ok
usb5/6 usb-kbd connected configured ok
usb5/7 unknown empty unconfigured ok
usb5/8 unknown empty unconfigured ok
usb5/9 unknown empty unconfigured ok
usb5/10 unknown empty unconfigured ok
usb6/1 unknown empty unconfigured ok
usb6/2 usb-storage connected configured ok
usb6/3 unknown empty unconfigured ok
usb6/4 unknown empty unconfigured ok
usb6/5 unknown empty unconfigured ok
usb6/6 unknown empty unconfigured ok
usb6/7 unknown empty unconfigured ok
usb6/8 unknown empty unconfigured ok
usb6/9 unknown empty unconfigured ok
usb6/10 unknown empty unconfigured ok
# cfgadm -c configure sata4/1
cfgadm: Hardware specific failure: Failed to config device at ap_id: /devices/pci@0,0/pci10de,375@e/pci1043,81e4@0:1

I didn’t get a chance yet to look at the differences between the two processors you mention, but if you find out anything it would be interesting to know.

Cheers,
Simon

Thanks for your very informative guide on this subject. It still seems to be one of the best guides on the web about building a home ZFS server even after two years.

I was wondering if you had any advice on choosing between the AMD Athlon X2 4450B http://www.newegg.com/Product/Product.aspx?Item=N82E16819103264 and the Opteron 1354 (Budapest) http://www.newegg.com/Product/Product.aspx?Item=N82E16819105211. The 4450B seems like a slightly newer model of the processor you have, with the same 45W TDP. If the Opteron has much more thorough data protection, maybe the extra $35 and 35W of power usage are worth it.

I remember that you did mention that “Opterons would be the best choice for the increased ECC protection” and was wondering if you had more information on this. My google-fu cannot reveal any information on AMD’s website about the different levels of ECC protection between the Athlon X2 and Opteron 1000 series. I did find this page: http://www.amd.com/gb-uk/Processors/ProductInformation/0,,30_118_8826_14266,00.html but it has no mention of whether the L3 cache is protected by ECC. It seems to me that the Athlon X2 family does not have a L3 cache at all, but maybe no L3 cache is better than a L3 cache with an unknown level of protection.

I don’t have a lot of time right now to give this a proper answer, but here’s a couple of points:

* I’m not familiar with the Phenoms, but you’d do well to research the power consumption/speed etc of these processors in more detail for the tasks you think it will be used for (storage, video streaming, encoding etc).

* With that many drives (15 or so) you might want to consider using RAID-Z2 (double parity), as you’ll have a lot of video data which you certainly don’t want to lose.

* You’ll probably want to consider use of a good SATA controller card like the Supermicro AOC-SAT2-MV8 card which can drive 8 disks, and it appears to be the favoured SATA card due to having the same driver that’s used in Sun’s ‘Thumper’ storage system (SunFire X4500), driving a Marvell SATA chipset.

* There’s also a SAS controller card that I have seen mentioned, the AOC-USAS-L8i, but I’m not sure of its compatibility, but from what I’ve seen, it looks like it should be compatible and perhaps an even better choice than the AOC-SAT2-MV8 card, and you can still use SATA drives with it — see this guy’s ZFS media server post:
http://zpool.org/2008/12/16/my-zfs-media-server

* For running an alternative OS inside OpenSolaris, you might want to take a look at the Zones / Containers / Branded Zones feature, as I think it allows you to run, for example, Linux inside the Zone and assign to it its own ZFS file system which it cannot write outside of. Whether this use of Zones would work for what you need, I’m not sure, but if you find out anything, I would be very interested to hear from you.

Hopefully this was of some use, and sorry I don’t have all the answers

Cheers,
Simon

I’ve settled on the ASUS M4N78 PRO (http://www.newegg.com/Product/Product.aspx?Item=N82E16813131382&Tpk=m4n78%20pro) for the motherboard, but I’m kind of utterly lost on which CPU to get. Do you have any idea if the ECC support was improved between Athlon X2s/Phenom X3/X4s and the Phenom IIs?

$67 (Phenom X3 2.1 ghz; 65nm) x3 8450
$112 (Phenom X4 2.2 ghz; 65nm) x4 9550
$121 (Phenom X4 2.3 ghz; 65nm) x4 9650
$121 (Phenom II X3 2.6 ghz; 45 nm + better cache) II x3 710

I’m kind of leaning towards X4 vs Phenom II X3 (more cores vs fewer but faster/more efficient cores), but I’m not sure if I actually need that much processing power. I’m building a ZFS server with enough space to expand to ~15 drives; at most it should only be streaming HD video to two separate places at a time while running Bittorrent/other P2P software and some video encoding. I’m hoping add in a capture card of some sort to give it minor DVR functionality, but I read that it’s difficult with OpenSolaris/FreeBSD; I might try running a virtual machine with MythTV to do that. Would this sort of usage benefit significantly from the extra core, or should I just stick with the Phenom II?

I’ve done some googling, but most of the comparisons seem confined to gaming performance and unanimously favour the Phenom II X3s over Phenom X4s, so any input you could offer would be greatly appreciated.

As ZFS on FreeBSD is a port from Sun’s ZFS implementation in Solaris, in theory it can never be at the same level of features/bug-fixes. I chose Sun’s ZFS as I wanted to have the reference implementation.

Over time, I’m sure the FreeBSD port will catch up with Sun’s version, but that may or may not be the case at present, so you’ll have to decide for yourself, as I’m not spending any time following FreeBSD status.

Others may be able to give you a more informed opinion on the FreeBSD ZFS port status, but be careful of over-optimistic evangelist “it’s perfect and has no bugs” talk.

Hope this helps.

Simon

Location: 3 (system board or motherboard)
Use: 3 (system memory)
ECC: 3 (none)
Number of Slots/Sockets: 4
Memory Error Data: Not Supported
Max Capacity: 4294967296 bytes

ID SIZE TYPE
65 62 SMB_TYPE_MEMDEVICE (memory device)

Manufacturer: None
Serial Number: None
Asset Tag: None
Location Tag: DIMM_B1
Part Number: None

Physical Memory Array: 64
Memory Error Data: Not Supported
Total Width: 72 bits
Data Width: 64 bits
Size: 2147483648 bytes
Form Factor: 9 (DIMM)
Set: None
Memory Type: 18 (DDR)
Flags: 0×0
Speed: 1ns
Device Locator: DIMM_B1
Bank Locator: Bank0/1

For the video-server, I went for the Aopen ES50C case again and for an AMD processor. I got an Asus M3A78-CM mobo, AMD 2.3 GHz Phenom 9650 (95W B3 stepping), 4 x 2 GB KVR800D2N5 (no ECC but around 7 euros/GB and after burning in and testing for weak bits, I’ll live with it if they screw a bit somewhere in a 700 MB movie), a 2-3 year-old 180 GB Deskstar for system and 8 x 1.0 TB SATA2 WD10EAVS green drives (around 70 euros a pop so I got a ninth as a quick spare in case of emergency resilver). I put things together in my spare time, last week.

Since the video is on the mobo and the drives are less than 10W each, I didn’t bother to change the 300W operating, 350W peak Aopen PSU, but I keep a spare Enermax on hand as insurance. I installed Solaris 10 u6 and replaced the PS2 keyboard for a USB one. I have to confess I’m a pig when it comes to storage, I never seem to have enough, so I would setup these 8 drives as RAID 5 to make for almost 7 TB usable space, but on the other hand the cautious little voice is whispering in my ear that the track record of these drives is very short and I should RAID 6 them until more historical data is available, especially considering the recent Seagate 1 TB bricking drives fiasco or saga depending on if you bought some or not, of last fall as described for example in: http://www.tomshardware.com/news/seagate-500gb-1tb-firmware-update,6867.html
I got one in September 2008 (for 180 euros) and the serial number checker on Seagate’s site says it’s OK even if the serial is among the suspect ones talked about on forums. Anyway unless some of your participants have a real strong argument for setting them up with one parity I’ll be going with two parity and take the 1 TB loss for a safer but only 6 TB usable pool. It won’t be enough to put the 1672 DVDs I burned of compressed avi, mpg and mkv material but it’s a start and later I can make another server for the uncompressed transport stream captures from my coolsat 7000 receiver that are waiting on 8-9 500 GB usb drives.

So right now i have a few questions for the nice people tuned on this blog:

1. What’s the best links and ebooks to check for recent solaris 10 configuration and setup info to get this little beast up and running without bugging you all forever with these specifics and those nagging wallbangers i will encounter on the way. Out of the circle of sun sites there is not much i could muster.

2. Search as I might, I couldn’t find an affordable USB2 DVD burner or even only reader that you can load with a pile of say 100 DVDs so that it sends the files into a folder on the server leaving you free to attend other less menial tasks. At about 10 minutes per DVD these 1680 DVDs would be done in less than 300 hours or about 2 weeks, and the DVDs can be kept as backups reducing the need for backups to other hard disks for the needed protection of data.

Paul in the snowy mountains north of Montreal.

An ‘NVidia GeForce 6200 Turbocache’ entry appears on the OpenSolaris HCL for video adapters here:
http://www.sun.com/bigadmin/hcl/data/os/components/views/video_adapter_all_results.mfg.page2.html

As the Asus card you mention appears to sound very similar, I suppose it should work.

Good luck with your build and it would be good to hear what you think of your system once it’s working.

Cheers,
Simon

I’ll just decide on RAM and PSU, then I’ll buy it!

Mats

I use an Asus EN6200LE card, but there was an Nvidia driver problem with S3 suspend/resume, but I believe it may have been fixed in SXCE build 104 or greater, which I will be testing soon (I have SXCE build 106 on a CD here waiting for me to try…).

Cheers,
Simon

I have looked at some of the cheapest nvidia/ati-cards, but the information about suspend-support is really scattered, if any at all. Did you have any cards in particular in mind?

Thanks,
Mats