fd0

Have you bought all of your hardware yet? If not, I’d recommend a couple of changes:

1) Motherboard: X8SI6-F. It’s like the X8SIL-F, but it has an onboard LSI SAS2008 (6Gbps) controller with 2 ports so you can skip the extra SAS controller card. It is a bit more expensive than the X8SIL-F, but should be less than the X8SIL-F + the AOC-USAS-L8i. It does only have 1 x8 PCI-e slot on it, tho.
2) Memory: Super Talent DDR3-1333 4GB/256×8 ECC Micron Server Memory – W1333EB4GM. It is cheaper and faster (you can generally find it for ~50 a stick on SuperBiiz and they frequently have discount coupons).

I have both of those in my home system and they appear to be working just fine.

-Colin

- 4k drives are emulated as 512-bytes sector for s****y Windows XP. Huge and catastrophic impacts on the performance under ZFS because of WD’s care for retarded customers -> I will use the gnop tips to create a virtual disk with 4k sectors as first of my pool + I could also use a modified zpool that lets me specify the ashift from 9 (bad) to 12 (good).
So there is at least two solutions for this issue. But I recognize we should not be constraint on using work-arounds, and WD should definitively release a non emulated firmware to its customers. BTW, tons of new HDDs are also concerned and are using this new method of partitioning, so it is not a Caviar Green only issue.

- TLER setting hardwired on OFF since one year. Apparently, there is no real impact under RAIDZ, in fact, TLER should be DEACTIVATED under a RAIDZ configuration ! so no issue at all, this is just a big confusion around

- Finally, the too many parking issue (load-unload cycles that are raising dramatically). Because it is an energy efficient drive, heads on Caviar Green are parking too often under a RAID configuration, slowing down the whole thing and decreasing dramatically the lifespan of the whole. Lucky we are, there is an utility that is still working on the WD website (WDIDLE3) to set the delay from couple of seconds to 5 minutes, which is way better. Problem solved.

So I will stick with the WD 2 TB EARS. Pity that the only real issue (the 4k advanced format), that is not restricted to the Caviar Green series, is still present and needs a work-around, but I don’t like other offers.

I have a OpenSolaris 2009.06 running, and currently planning on upgrading both HD and software (new Oracle Solaris Express ? other ?). The old machine will serve as a backup machine for my data.

Here are the specs I selected for the various components.

[*] Intel Xeon L3406 (Just seen you were stuck with the E3110, but this one is 32 nm and max 30 Watts compared to 65 W …) http://ark.intel.com/Product.aspx?id=47555
[*] SUPERMICRO X8SIL-F – micro ATX – Intel 3420 – LGA1156 Socket (micro ATX, good motherboard brand, ECC memory, 6 SATA and 2 Intel GB Ethernet ports) http://www.supermicro.com/xeon_3400/Motherboard/X8SIL.cfm
[*] Crucial 8 Go (Kit 2x 4 Go) DDR3-SDRAM PC8500 ECC CL7 – CT2KIT51272BA1067 (Crucial, ECC …)
[*] Lian Li PC-V354B – mini desktop (very small factor, but seven 3.5″ HDD drive bays inside + one 5″1/4) http://www.lian-li.com/v2/tw/product/upload/image/v354/flyer.html
[*] TWO * Corsair Force Series F60 – SSD 60 Go 2.5″ Serial ATA II (mirrored rpool, to be inserted in a 5″1/4 rack and connected to the on-board SATA controller)
[*] Supermicro AOC-USAS-L8i 8 Port SAS RAID Card (for managing the tank pool)
[*] SEVEN * Western Digital Caviar Green 2 To 64 Mo Serial ATA II – WD20EARS (9 TB tank in RAIDZ2 mode, no spare for the moment)
[*] SAS to SATA cables, Seasonic fanless 400W PSU, Noctua fans etc…

Looking at your proposal, I find really lacking the ECC capability, IMHO. I mean, you want to install a fireproof FS, but withoutECC RAM, which is mandatory for most serious server systems. And you already have a Xeon type processor anyway.

I am currently buying components, planning to be done by mid March with a running machine.

Cheers

Julien

Of all the guides and sites of looked at over the past few weeks, yours appears the most easy to understand:) I have a scenario for you (and anyone else who’d care to reply):

After 16 years of being an IT Engineer in a Microsoft-based environment, I bought an iPhone 3G a couple years ago and have given up on Microsoft for personal use. My home is 90% Apple with a sprinkle of ‘other’ here and there for budgetary purposes. I’ve run out of storage space on a couple of external USB drives and am about to build some type of file storage system. Here is my current setup at home:

Main System – iMac 21.5″, 2.8Ghz C2Duo, 4GB, 500GB.
– Running iTunes, Plex Media Server, Air Video Server

Secondary Systems – Apple TV2, LG Blu-Ray (DLNA), iPhone 4, iPad
– Running Plex Client on all but LG Blu-Ray for media streaming in-house and over internet via wi-fi/3G.

Backup/File Storage – (2) 1TB External USB HDD’s.
– Uses: Time Machine Backup of iTunes, iPhoto, Documents and File Storage for Media Sharing.

Here is what I would like to do:

- Add an 8-10TB File Server/NAS to my network.
- Compatibility with both OS X and Windows for streaming.
- RAID 5 or 6 Redundancy (know RAID5, learning about 6).
- Ability to expand storage later if necessary without complete rebuild.
- Move remaining media off iMac and external drives to run off server, create iTunes Server.
- Use USB External’s for Time Machine backup’s of documents, photos and home movies.

Here is the equipment I’ve chosen based on existing and budget:

- Existing Intel Xeon E3110 Dual Core @ 3.0Ghz, LGA775
- New Gigabyte GA-G31M-ES2L Mobo
- New Crucial 4GB DDR2 800 Memory
- New (2) Promise SATA300 TX4 4-Port PCI SATA II Controller
- New (6) WD Caviar Green 2TB 3.5″ HDD’s
- Existing Full ATX Tower Case with 12 total drive bays
- Existing Antec 500W Phantom PwrSupply

Based on the information above, is your guide up-to-date, any recommendations/changes as well as any additional assistance to a novice Unix/Linux/Solaris user? Oh and I have to stay under an $800 budget with the Tower, PS and CPU already purchased/not included.

Please advise…this looks to be the best option and your guide to be the easiest to follow.

Brad

Without knowing the exact circumstances & reasons for the errors you are seeing, it is difficult to make any useful suggestions. It sounds quite likely there is some possibility of a hardware error. How long did you have the pool in operation, and how often did you perform a scrub on the pool? With more frequent scrubs you will likely see signs of read/write/checksum errors occurring long before serious failures, allowing you to swap drive(s) showing signs of trouble. For home systems, a monthly scrub should be sufficient.

Personally, having run two ZFS NAS systems for the last 3 years, I have never lost any data, and never saw even one read/write or checksum error within the main data storage pools, although I did see some errors occurring in my mirrored boot SSDs: http://breden.org.uk/2009/09/02/home-fileserver-handling-pool-errors/

Indeed, backups are highly recommended.

It’s quite possible that your system can be recovered, so before doing anything with your system, I would strongly suggest creating a post describing what happened to your system on the OpenSolaris ZFS forum here: http://opensolaris.org/jive/forum.jspa?forumID=80

Cheers,
Simon

I bought a 2 TB USB external hard drive and I am going to start doing regular backups of the entire filesystem in case this ever happens again. I guess I got a little overconfident figuring with raidz2 that I would have to lose 3 drives before data corruption.

Thanks!

From the message it appears that the file shown has unrecoverable errors. The remedy is to recover the file from backups, if you have them.

If you don’t have a backup then you could try to locate an earlier uncorrupted version of the file, using the snapshots. Look in the .zfs directory to find the file. See the ‘Checking the snapshots’ section of this post to see how to reference individual files within various snapshots of a given file system.

Hope it helps.

Cheers,
Simon

pool: tank
state: ONLINE
status: One or more devices has experienced an error resulting in data
corruption. Applications may be affected.
action: Restore the file in question if possible. Otherwise restore the
entire pool from backup.
see: http://www.sun.com/msg/ZFS-8000-8A
scrub: resilver in progress for 12h0m, 53.91% done, 10h15m to go
config:

NAME STATE READ WRITE CKSUM
tank ONLINE 0 0 2
raidz2-0 ONLINE 0 0 2
c7t0d0 ONLINE 0 0 0 98.3G resilvered
c7t1d0 ONLINE 0 0 0 98.2G resilvered
c7t2d0 ONLINE 0 0 0
c7t3d0 ONLINE 0 0 0
c9d0 ONLINE 0 0 0
c9d1 ONLINE 0 0 0

errors: Permanent errors have been detected in the following files:

tank/itunes@zfs-auto-snap:daily-2010-10-11-00:00:/iTunes/Album Artwork/Cache/7C7DE397DD748486/07/09/00/7C7DE397DD748486-872B56E2544E1097.itc

However, I agree it’s a pity that he has axed OpenSolaris. Luckily, the community will keep the OpenSolaris code going with OpenIndiana.

ellison = nazi

Thanks and glad you managed to make your own NAS!

If I were you, I would upgrade to build 134 of OpenSolaris, until the independent OpenSolaris forks are properly under development — they’re still in setup stage at the moment from what I’ve seen. Build 134 was the last 2009.06 version released and was to become the OpenSolaris 2010.xx that never got released. I use build 134 and it has worked fine for me. I use CIFS sharing to Mac OS X and have not noticed any problems.

Once OpenIndiana has a few releases that work, I will probably try upgrading to it — see here:
http://openindiana.org/download/

I would stick with OpenSolaris or its forks (OpenIndiana/Illumos), as you have:
1. The reference implementation of ZFS (less risk to your data in theory)
2. OpenSolaris has BE’s – boot environments – that easily let you create multiple bootable working versions of OpenSolaris – take a look here for an idea of what I mean: Home Fileserver: Mirrored SSD ZFS root boot.

Cheers,
Simon

Thanks for sharing your work and effort to date.
With the info, I’ve been able to build my own – much appreciated.

I am wondering (with the demise of Opensolaris) if you are considering migrating to another platform with a bit more life in it?

I’ve installed 2009.06, which apparently has known issues with CIFS (just dies, requires poweroff/on to fix) and my (personally) NFS isn’t great for any sustained copies over 500MB.
Looking into the CIFS (most used by me) issue, the suggestion is to “upgrade” to the Dev build…

I am considering it (it looks pretty easy).. and or migrating the OS to a freeBSD-based distribution instead.

thanks,
Matt

I would like to thank you for making this tutorial. It has been very very good and has helped me many times in my setup of ZFS and Opensolaris.

Cheers!

ps. Can you also send me a PDF of all posts and comments thus far?

Thanks for the request.

What would you like to see in the PDF? All of the posts, with comments too?

Cheers,
Simon

thanks for your information on OpenSolaris and your Home Server Setup and also for answering all the questions from the comments. So I hope you can also answer my little ones.

I can think of a scenario with one system drive and 4 data drives bound to a raidz1. Now my two questions:

a) How should one backup the system drive? Doing snapshots to the raidz1 pool and in case something bad happens with the motherboard or the system drive put everything back to a new system drive? How would this scenario work in real life?
Is it possible to snapshot the whole system drive and than to clone it back to another disk so that it is ready to boot? What needs to be done to achieve this?

b) Think of one part of the raidz1 pool is dedicated as time machine backup place for a mac and thus exported via iSCSI as a block device to the Mac system. Also deduplication is switched on. What needs to be done to recover the Mac if its disk fails. How do I get the backup back on the new Mac disk. I think I can’t clone it directly back to the new Mac disk?

Thanks for your help and keep up the good work,

Jan.

If you’re more comfortable with stable, infrequent releases, try Solaris 10:
http://www.sun.com/software/solaris/get.jsp
http://www.sun.com/software/solaris/

Choosing which version of Solaris to use is always a tradeoff: (1) OpenSolaris=frequent releases containing new features under development or (2) Solaris=infrequent releases, but heavily tested and rock solid.

Cheers,
Simon

Simon, this unfortunately was very bad advice and I lost several days by installing and wrestling with Open Solaris 2009.06 for use as a file server.

Both smb/server and network/samba are _badly_ broken and as I later learned the hard way Sun admins are fully aware of the situation but did not publish any warnings on the download page.

With all the problem reports google brought up imho Open Solaris is the testbed for Sun and success with such distributions heviliy depending on luck and local weather conditions.

My conclusion: Unless you are prepared for a long and ugly struggle: Avoid Open Solaris!

someone

Second i would like to configure an ftpserver so i can access and transfer files from outside my home network. I really havent found a good guide to do this….
The nas is connected to a router with the firewall enabled, is this enough?
Should i configure the “ipfilter” in opensolaris too? Is there a guide somewhere for this?
I have tried to look at man pages but i havent found anything useful…
As you can see im a unix beginner. I think Opensolaris is a great OS, but its not very userfriendly and
information on how to set up certain bits is not really abundant.
Maybe im looking in the wrong places?

Im now configuring/administrating/checking health on this machine via PuTTY SSH (which is pretty slow)
and sometimes via VNC
Isnt there a faster and easier way to do this??
What web interface alternatives are there?
I have found something called Webmin, is this the best one with the most features that you may need?

Your blog has helped me a lot already, but maybe you can point me in the right direction regarding my
questions?

Thanks a lot!!
Gurkman

If you have basic Linux knowledge then you should find it relatively easy to use OpenSolaris too. You can find the answer to any admin problem online: Google, OpenSolaris forums and blogs etc.

For using Active Directory and Solaris, see the AD link at the top of this page.

Regarding bottlenecks, the network will be the bottleneck when transferring big files, but even with commodity Gigabit ethernet you should get speeds of around 50 MBytes/sec, which should be sufficient. If you need serious speed, look at 10GbE or Direct Attached Storage (DAS) using tech like InfiniBand, which is used with an HBA, and is often used by video editing software to connect to moderately fast storage devices like this, which easily reach speeds of around 500+ MBytes/sec sustained using 8 drives in a redundant array.

Regarding running 2 boxes at home, you can probably ditch Windows and use some Solaris directory service, if you need it, but I didn’t look at this.

Good luck,
Simon

I have a Home Server with Windows 2003 Server (Domain Controller) with Hardware-RAID5. There are about 4 Windows Clients connected to this server.
Primary use of this server is file storage (huge collection of DVDs, images, mp3 collection, pdf scans and documents), some applications and web server.

I’m a windows guy and have only minor knowledge in linux/OS, but I REALLY WANT the superior ZFS.

Is it possible to run OpenSolaris with ZFS and within this system a virtualized Windows 2008 Server (with Active Directory)? I’d like to setup OS and ZFS as a (hopefully) stable “install-and-forget” system. Windows Server would be see the ZFS-Pool as a huge virtual drive and shares this drive to its Windows clients. So all my Windows clients would be connected to the Windows Server only. They (and Windows Server) don’t know of any OpenSolaris Server.
With this scenario I could continue to work with Windows Server, but can trust all my datas to the ZFS.

Is this possible? If yes, is there any performance bottleneck when transfering huge files from the clients to the server (or vice versa)?

Most time, my home server is idle and not very busy. (2 concurrent users)

For my home environment, I don’t want to run two dedicated server (OS for storage and Win2008 for Domain), so I really would like to virtualize it.

What’s your opinion?

And thanks for your great site. Lots of very useful information!