A Home Fileserver using ZFS

Hi,
very interesting project – can you tell me which version of solaris did you use ? there seems to be Solaris 10, Open Solaris, Solaris Developers Edition …. Which one is the most suitable for a stable, reliable fileserver ? I’m after data integrity (zfs), redundancy (raidz) and something simple to admin that “just works!”

thanks
CW

Thank you for this. Your walkthrough persuaded me to give ZFS a try over unRAID and (heaven forbid) WHS.

One of the few things that originally had me leaning towards a WHS machine was the availability of a simple plug-in for implementing off-site backups using Amazon’s S3 service.

I found this and plan to give it a go:
http://developers.sun.com/solaris/articles/storage_utils.html

Have you tried something similar? I appreciate the information on incremental snapshots.

ZFS seems to be far and away the most powerful means for implementing a home NAS.

From what I’ve seen of people using Amazon S3 for off-site Windows Home Server backups, they pay only a few US Dollars a month for the service.

This page has the info:
http://www.amazon.com/gp/browse.html?node=16427261

They charge for the amount stored and the amount transfered. I’d have to imagine that snapshots are generally very small for a system used this way (very few deletions and/or changes) so the transfer costs would be minimal.

I don’t know how Amazon S3 compares to other services, but it seems to be the go-to service for WHS off-site backups. Good luck!

I’m surprised no one considers either NexentaOS or NexentaStor, a prebuild NAS using ZFS.

Hey Simon, I’m still playing with ZFS on vmware while my hardware is on order. I like to familiarise myself with it as much as possible. There’s just one thing that I don’t seem to be able to find an answer to anyway and I wonder if you can help.

I’ve got a zfs shared with smb, not samba as i understand it but CIFS. SMB authentication through pam is working fine and I can create files on the share perfectly. The only problem is that after a file is created, it is assigned no permissions whatsoever. I know that using samba I would use the “create mask” option, but seeing as this is CIFS I’m a little stuck. Google hasn’t brought me much success and the osol-discuss mailing list hasn’t thrown any answers either.

Thanks

I am going to try setting this up but don’t have much experience. My question is… can I use a bunch of different size drives for ZFS or do they all have to be the same. I will be using this to serve media throughout the home as well as store work docs so that I can access from any computer. Ie. a pool for each of these: movies, music, pictures, home videos, work
My primary goal is to be able to serve these up quickly over my gig lan but I also don’t want to lose the data. I have 3 750gb disks and then I have 4 that range from 160 to 300. Would I just setup two different sets of pools? Ie. use the 3 750gb just like you set it up and then create pool using the 4 disks. sorry if this is really basic but I just don’t understand as I am very new to this.

Thanks

Hi JV,

the short answer is it depends what kind of pool configuration you want. If you want to create a simple RAIDZ1 configuration then it’s best to have the same size drives, probably the same make and model too — i.e. I create a pool comprised of a single RAIDZ1 vdev that uses three 750 GB drives.

Or, if you had four drives, say two 500 GB drives and two 750 GB drives then you could create a pool composed of two mirrors, one mirror using the two 500 GB drives, and the other mirror composed of two 750 GB drives.

The problem with trying to make vdevs from multiple different sized drives is that each disk will have the capacity of the smallest drive. For example, if you had three drives — a 160 GB, a 320 GB and a 750 GB drive, and you formed a RAIDZ1 vdev from these, ZFS will make all the drives 160 GB in size, so you’d only be able to create a vdev from these which has a capacity of 480 GB (3 x 160 GB).

If you have a bunch of odd-sized drives, then perhaps you can use them to make a backup pool, and use no redundancy, thereby getting the full capacity of each drive. This is exactly what I did to make use of a bunch of old SATA drives I had — and I made a backup server from all these old drives.

Hopefully that answers your question.

Sorry, been a long day, and re-reading your question I might make the following suggestions:

1. Make one big pool using the three 750 GB drives, probably best to use a single RAIDZ1 vdev — i.e. zpool create mypool RAIDZ1 disk1 disk2 disk3
2. Once the pool is created, I would create 5 file systems within this pool for each of: movies, music, pictures, home videos, work. You can then assign properties to each file system independently. For example, you might decide to have only one copy of each file within the movies and music file systems, but you might decide to create 2 copies of each file within the pictures, home videos and work file systems, as they are content created by you which cannot be recovered if data is lost. For this see the ‘set copies=2’ attribute for the file system. Some people prefer to create 2 disk mirrors for personally created content, as each disk in the mirror is an exact copy of the other one.
3. See my comment above regarding using 4 different sized disks. Basically, you have 2 choices if you include a vdev in your pool comprising these 4 different-sized disks: (1) create a RAIDZ1 vdev but then ZFS will use the smallest disk size for all 4, or (2) create a vdev with no redundancy and be able to have the full capacity of all the drives added together — but the problem is that if there’s a an error you will probably lose the affected data permanently! For this reason, like I did, it may be best to do this (no redundancy) only for a backup server/pool, where you have the original data in the main server’s pool. It’s your choice.

Hi Simon,

you posted a problem concerning using “cp” to copy files at http://www.opensolaris.org/jive/thread.jspa?threadID=59201

Are there any new information about that problem? Did you try the latest snv_91?

Best regards
Chris

Hi Chris, no I have no more news on that bug as I’m still on snv_87. However it looks like bug 6669134 was fixed in snv_90, so it’s possible that the bug has been fixed now. However, there were a number of similar-looking bugs that I listed in that post, and some of them have not been fixed yet, so it’s quite possible that the bug still remains in snv_91.

If you try snv_91, it would be great if you can confirm whether you see the bug. Likewise, when I upgrade, I will note here if I find the bug has been fixed.

Cheers,
Simon

Great write up.I appreciate it.I’ve been planning a new NAS set up and this really looks good.

Al

Hi Simon,

I made an upgrade on my OpenSolaris 2008.05 machine. The kernel is now snv_91 and the problem is still not solved. Too bad!

Best Regards,
Chris

I openend a bug. http://defect.opensolaris.org/bz/show_bug.cgi?id=2366
I already got a hint how to debug this behaviour. I hope to get it done later today.

Chris

My question is are you using a full solaris install or something that only includes the ZFS, NFS and CFIS functionality maybe some iSCSI? It seems like all the home file servers are using the bloat approach, I’m curious if anyone has gotten it down in the 128MB ~ 64MB range for a true appliance style.

Hi Dave, I’m using a full, standard OpenSolaris install — Nevada, build 87 currently. I believe Nexenta may be what you are looking for, but be aware that you only get a free version capable of running a 2TB array, last time I looked. If you will ever need to store more than 2TB then you will be paying to use this software. That’s one of the main reasons I chose to use the standard version of OpenSolaris.

Okay Simon. Looking forward to hear from you.
A note on snv_93: I barely use it, but the ZFS web administration interface seems to be broken (again!). But ZFS boot in the installer is very nice! 😉

Have a nice weekend!

Take care,
Chris

Simon on June 26th, 2008 at 18:40 wrote:

> Hi JV,
> the short answer is it depends what kind of pool configuration you want. If you want
> to create a simple RAIDZ1 configuration then it’s best to have the same size drives,
> probably the same make and model too — i.e. I create a pool comprised of a single
> RAIDZ1 vdev that uses three 750 GB drives.

True.

> Or, if you had four drives, say two 500 GB drives and two 750 GB drives then you could
> create a pool composed of two mirrors, one mirror using the two 500 GB drives, and the
> other mirror composed of two 750 GB drives.

Using OpenSolaris / Solaris / SXCE / etc. you could partition the drives into 250GB chunks
and then RAID-Z2 / Mirror the pieces back together. If you do it correctly then the different
chunks on different drives / controllers will each protect one and other.

Depending on your configuration you can add the 500GB drives to the 750GB drives and then
mirror / raid them and end up with 1250GB of drive space with twice the speed (IOPS) or
take the total of (2 * (500GB + 750GB)) 2500GB and use a RAID-Z2 9+2 to get a huge MTTDL
and a lot of drive space.

Lots of ways to slice it 🙂 . Partitioning improves defrag times, it is not evil.

Joel

Hi Simon,

if you use CIFS as a service don’t update to snv_93!
http://www.opensolaris.org/jive/thread.jspa?threadID=65996

I haven’t tested CIFS that much, but it seems to be a big problem.
So if you not already updated to snv_93, wait for snv_94! 😉

Regards,
Chris

snv_94 was released today! Hopefully the bug is fixed! 😉
I will install snv_94 on my filer, too.

Best regards,
Chris

The bug is gone. 🙂
http://www.opensolaris.org/jive/thread.jspa?threadID=65996

Let’s try snv_94! 😉

Regards,
Chris

Thanks Brian. Not sure about the web interface. Honestly, ZFS is so easy to setup with the command line, that I didn’t consider the web interface necessary. Good luck with your new system!

Hi Matt,

Fantastic news. If you could send me some summary blog text, I will use it to create a new post, and will link to your article from there to give the full details that you have written.

Thanks a lot.

Simon

Hi Bryan,

In fact it’s worse now for NexentaStor: they’ve reduced the limit to 1TB now for the Developer Edition, or instead you can choose the Free Trial Edition with no capacity limits stated but you have a 30 day time-limit. As the scope of this post is for a home fileserver/NAS, NexentaStor is not suitable, as the annual ongoing costs are prohibitive — it’s meant for businesses.

Personally, I had no use for NexentaOS/NexentaCore as all that I needed was already in OpenSolaris, which is also free. Also, the APT features have now been included in OpenSolaris 2008.05, known as IPS (Image Packaging System), which you can use to download and install free security fixes. However, I believe that you have to pay for the ability to have full use of IPS for updating other packages (http://www.sun.com/service/opensolaris/index.jsp), and this appears to be an ongoing $324 payment. Again, as the scope of this post is for a home fileserver/NAS, this is unacceptable. For businesses, that may be another matter.

However, if you’re saying that NexentaCore/NexentaOS are free, have no restrictions of any sort, and allow easy updating via APT then it could be interesting.

But I do like the fact that by using the basic vanilla OpenSolaris, I know exactly what I’m using (which version etc), and can fire off questions to the great enthusiasts in Sun’s OpenSolaris ZFS forums, for example, or general driver questions, and there’s fewer external things to depend on. Anyway, for what I’ve needed here, this approach has worked fine so far.

Simon

Have you heard about the SUN Enterprise storage server 7000?

An ordinary disk has like 200-300 IOPS. There is an extremely fast Savvio disk that exceeds 400 IOPS. Tradiditionally, you buy lots of 15000RPM disks to reach high IOPS. The SUN 7000 storageserver can reach 250,000 IOPS (in extreme cases). Watch the extremely cool video on post marked “31 dec 2008”:
http://blogs.sun.com/brendan/

The thing is, the 7000 reaches sick IOPS because it uses many SATA 7200 RPM disks + ZFS + SSD drives as a cache. It is the SSD drives that acts as a cache, that does it.

Here is the point: you can add a SSD drive to your existing ZFS raid as a cache and reach high IOPS. ZFS will administer everything for you, you just type in:
#zpool add cache device
or something like that.

Imagine 10 SATA discs in raidz2 (similar to raid-6) and one or two SSD drives as a cache. Each Vista client reaches ~90MB/sec to the server, using Solaris CIFS and iSCSI. So you want to use iSCSI with this. (iSCSI allows ZFS to export a file system as a native SCSI disc to a desktop PC. The desktop PC can mount this iSCSI disk as a native SCSI disk and format it with NTFS – on top of ZFS with snapshots, etc. This is done from desktop PC bios).

Now, you install WinXP on the iSCSI ZFS volume, and clone it with a snapshot. Then you can boot from the clone, with the iSCSI volume on a desktop PC. Thus, your desktop PC doesnt need any hard drive at all. It uses the iSCSI volume on the ZFS server as a native SCSI disk, which has WinXP installed.

This way, you can deploy lots of desktop PC in an instant. Using the cloned WinXP snapshot. And if there are problems e.g virus, just destroy the clone and create a new one in one second.

With 10 discs in ZFS raid, you maybe reach ~600 MB/sec. And then, you add SSD disk as a cache, and the IOPS ramp up very much. This solution is very very secure (ZFS + raidz2) and very very fast (SSD disk provides lots of IOPS) and very very cheap (7200 SATA discs, no expensive HW raid) and very very easy to manage (ZFS + iSCSI + CIFS is extremely easy). What do you think about this solution? Quite extreme?

SUN thumper with 48 discs, reaches 2GB/sec write speed, and 3GB/sec read speed. You surely need one or several 10Gbps NIC on your server.

Hi Kebabbert,

Yeah, that SUN Enterprise storage server 7000 gear looks pretty nice stuff for a business.
Do you mean the funny video in this post?: http://blogs.sun.com/brendan/entry/unusual_disk_latency where he shouts at the array?
Did you try building a setup where Windows boot image is stored on the ZFS fileserver?

Cheers,
Simon

Cheers Simon,
Another thing you may want to touch on: what if the mobo dies? Can one just connect the sata disks to another board, install Solaris, and Solaris will recognise the ZFS structure om the disks? That would be the ultimate security.
My worry with raid is: the raid controler dies and the new one knows nothing about the raid’ed disks and just prompts to partition the “empty” array.

Thanks again for your insight 🙂
Chris

Hi Chris,

No problem if the mobo dies, just connect up the drives that form your ZFS pool to a new compatible system, and the ZFS pool drives will automatically be recognised on next boot — ZFS records a special signature on the drives so the OS can recognise them. If you plan to do a graceful transfer of pool drives to another system/OS install, then you can type ‘#zpool export pool_name’, shutdown, disconnect drives from old system, install into new system which has the OS installed on another drive, and then just type ‘#zpool import pool_name’.

In fact it gets even better — as ZFS is endian-neutral, you can transfer ZFS pool drives from any-endian (little-endian / big-endian) system. When I first experimented with ZFS it was in a Mac Pro. Later, I moved the drives into a new home-built system running Solaris and used the same pool without any changes, and all the data remained available 🙂

So far, I have my OS installed on an old IDE drive outside the pool, and so OS and data are completely separate. If the boot drive dies, I’ll just chuck another one in the case and install Solaris again and be up and running very quickly.

As Solaris is open source, and there is no hardware RAID controller to worry about, your data is about as safe as it can ever be 🙂

Cheers,
Simon

Simon: I have to add my voice to the chorus of thank yous. OpenSolaris intriged me for a home file server system and your articles do a good job of explaining of the possibilities and what to do.

While I realize you did not write the article as an invitation for support questions, it would be appreciated if you or someone could clarify some ZFS points for me.

Is ZFS only meant for high end use cases? I thought one advantage to using it was that you can add drives to a storage pool and it all becomes transparent to the user, just more storage space. So for example, four or five small hard drives could be connected together and it would act like one large drive. (One advantage of this would be that a very large file might be bigger than one of the smallest drives, but under ZFS that would be OK.)

Now I wonder if all my assumptions were wrong. It seems that one can use ZFS in a way that is great for data recovery but drastically reduces the storage space (RAID-Z, mirroring, etc), can use it to pool together many disks (as mentioned above) giving you great storage abilities, but the failure of one drive ruins all the data, or you can use it in a “normal” way (one disk=one pool) giving no
advantages at all.

Is there something I am missing about ZFS? It seems that there are no advantages unless one cares soley about redundency and has unlimited resources for disk drives. That is to say not really a home file server. (Not to say that just because the server is in the home does not make the data unimportant.)

Simon:

Thank you muchly for responding. To read “ZFS is all about data integrity” puts more of what I learned in context. It is all very impressive and probably a no-brainer for use in large data centres. I think I still have to think about it for home use though. This is a home backup server. Going to it to retrive files means something terrible has already happened. While something terrible could happen twice simutanouslty it is very unlikely and stuff so important that one must take such a castatrophe into consideration would be archived onto optical discs and stored away from the computer. However the fact one can spend the extra money on hard disks and get such a great level of protection is appealing just to say I did it.

Thank you again for responding to my post. I do have a question still though. If I have a “a collection of old, different-sized drives to form a large backup pool” that uses no redundancy like you do in one case and like I am contemplating and one of those drives dies do I loose all the data in the pool or just some of it?

(Also, I assume redundency could not be added to that pool latter as the disk sizes affect the set-up of RAID-Z and other redundency schemes. I suppose when I can obtain more disks I could set-up a new pool and copy the data over.)

Thanks Duncan.

My first question would be to see why you want to use iSCSI, when you probably just need a simple CIFS share, or Samba if you use a Linux client and it doesn’t have true CIFS client support.

If you think you really need iSCSI, then bear in mind that an iSCSI volume is a raw block device, so once the initiator connects to the target, you’ll need to format it using the file system you wish to use.

Hope this helps and is correct, as I had to cast my mind back a year to my experiences with iSCSI.

For my media centre/HTPC I am using a CIFS share of a standard ZFS file system, as described here:
http://breden.org.uk/2009/05/10/home-fileserver-zfs-file-systems/

Cheers,
Simon

Hi Adrian,

As your data was not on the boot drive and in a separate storage pool, this should be easy to regain access to your data. Assuming your storage pool was called ‘tank’ then type:

# zpool import -f tank

You will probably need the -f option as the pool was never exported before the system motherboard and boot drive died.

If you can’t remember the name of your storage pool, no worries, just type:

# zpool import

and Solaris will list all available pools.

When ZFS imports the pool, it should automatically remount all ZFS file systems that the pool contains. It’s possible you might get some warnings/errors if you changed mountpoints if they don’t exist, but you’ll easily be able to fix those problems if you see them.

Also, as you will probably need to re-create your Solaris user(s), you’ll need to see which user ids and group ids you used within your file systems and recreate the users and groups with the same ids. Something like this:

# groupadd -g 501 fred
# useradd -g fred -u 501 -s /bin/bash -d /export/home/fred -c fred-flintstone -m fred
# passwd fred

Explanation of useradd parameters used above:
-g fred: adds user to primary group ‘fred’ (which has groupid 501)
-u 501: creates the userid 501 for this user
-s /bin/bash: assigns the default shell to be bash for this user
-d /export/home/fred: defines the home directory
-c fred-flintstone: creates the comments/notes to describe this user as required
-m: creates the home directory for the user
fred: this is the login name for this user

Hope it helps.

Cheers,
Simon

Thanks Miles, so you’d say there may still be problems associated with the AOC-SAT2-MV8. That’s a pity, so if I choose this card I’d better be sure to check the current situation from the solaris forums to see what people are finding.

When you say ‘at first the driver crashed all the time’, was this recently or a long time ago, as I believe there were many problems with the standard ‘SAT’ card (i.e. AOC-SAT-MV8), but that many of these problems were fixed in the later ‘SAT2’ card. I saw that info here, and he also says that the AOC-USAS-L8i (LSI megaraid chips?) would be his next choice — see here:
http://zpool.org/2008/12/16/my-zfs-media-server

He says:

This card [AOC-SAT-MV8] has had a lot of problems with ZFS and has been discontinued by Supermicro in favor of the AOC-SAT2-MV8. Whenever I expand beyond six disks, I’m going to skip straight to the AOC-USAS-L8i using mini-SAS to SATA cables.

I will take a look at your Automounter notes next. If I remember, the reason I used CIFS sharing instead of NFS was because of lousy write speeds (5MB/s with NFS instead of about 40MB/s with CIFS (50MB/s now with OpenSolaris 2009.06 and OS X 10.5.7 using 1 x GbE)), but maybe this has been fixed now. I think it was because of NFS using synchronous writes if I remember correctly. These quoted speeds were achieved writing from a Mac Pro to the ZFS file server.

Thanks a lot for the info. I have created a new thread on the Solaris forum with more questions/info — see here:
http://www.opensolaris.org/jive/thread.jspa?threadID=106210&tstart=0

Hi Mads,

This sounds like a good idea for a new blog post — thanks for the idea 🙂
Quick rundown: with ZFS you have the concept of a storage pool. The pool is comprised of one or more vdevs. A vdev is a collection of drives in a particular configuration — e.g. mirror, RAID-Z1 or RAID-Z2 etc. Currently a vdev of type RAID-Z1 / RAID-Z2 cannot be expanded, but Sun say they plan this functionality for the future (see http://blogs.sun.com/ahl/entry/expand_o_matic_raid_z). However, you can expand your pool as much as you like, by simply adding new vdev(s) when you need to, without any problem.

Knowing the above, for now, it’s best to decide how much space you’ll need and build in plenty of storage up-front. Just as an example, it is possible to break a 3-drive RAID-Z1 and recreate a 4-drive RAID-Z1, for example, as you can see here:
http://breden.org.uk/2008/09/01/home-fileserver-raidz-expansion/

Cheers,
Simon

How much do you want to spend on your storage in total? In this price I mean just the controller and the drives. And how much usable storage capacity do you want? And what level of redundancy do you want — i.e. how many drive failures do you want your system to survive?

You don’t need RAID cards for ZFS, quite the opposite. ZFS controls everything, and so you should use a JBOD setup. But you might want to buy a SATA controller to control your array of cheap SATA drives.

If I were you, I would assume that RAID-Z expandability is not high on the ZFS developers’ priority list, as they have stated this openly, and so I would just assume that you need to create an array with sufficient capacity for your foreseeable needs.

Personally I would not trust RAID level 5 for anything. AFAIK, RAID 5 does not use block checksums and so it has no way of knowing if there is ‘bit rot’ when it reads data back. RAID level 5 writes parity data to allow failed drives to be rebuilt, but if it can’t tell you that some bits in a file have been flipped then what good is it to anyone? Check out various CERN and Google reports on parity errors if you need convincing further.

ZFS was designed with the idea that you should never trust drive hardware, and so that’s why it has end-to-end data integrity built in from scratch, to help (1) be sure that what gets written and read back is actually what you told it to write, and (2) immediate detection of ‘bit rot’ on reading back of a file or during regular weekly/monthly ‘scrub’ operations where ZFS will read back ALL the blocks of every file in the storage pool and will check them against their block checksums, and if a difference is detected, ZFS will fix the corruption immediately from available parity data, assuming the vdev containing the corrupted file has redundancy — i.e. mirror, RAID-Z1 or RAID-Z2.

Hope this helps.

IMHO OpenSolaris 2009.06 using ZFS is very user-friendly, and you’ll know you’re using the standard ZFS distribution. Good luck.

Hi Rick,

If you have basic Linux knowledge then you should find it relatively easy to use OpenSolaris too. You can find the answer to any admin problem online: Google, OpenSolaris forums and blogs etc.

For using Active Directory and Solaris, see the AD link at the top of this page.

Regarding bottlenecks, the network will be the bottleneck when transferring big files, but even with commodity Gigabit ethernet you should get speeds of around 50 MBytes/sec, which should be sufficient. If you need serious speed, look at 10GbE or Direct Attached Storage (DAS) using tech like InfiniBand, which is used with an HBA, and is often used by video editing software to connect to moderately fast storage devices like this, which easily reach speeds of around 500+ MBytes/sec sustained using 8 drives in a redundant array.

Regarding running 2 boxes at home, you can probably ditch Windows and use some Solaris directory service, if you need it, but I didn’t look at this.

Good luck,
Simon

> Simon on September 2nd, 2009 at 15:44
> IMHO OpenSolaris 2009.06 using ZFS is very user-friendly, and you’ll
> know you’re using the standard ZFS distribution. Good luck.

Simon, this unfortunately was very bad advice and I lost several days by installing and wrestling with Open Solaris 2009.06 for use as a file server.

Both smb/server and network/samba are _badly_ broken and as I later learned the hard way Sun admins are fully aware of the situation but did not publish any warnings on the download page.

With all the problem reports google brought up imho Open Solaris is the testbed for Sun and success with such distributions heviliy depending on luck and local weather conditions.

My conclusion: Unless you are prepared for a long and ugly struggle: Avoid Open Solaris!

someone

May I ask you for a pdf with your complete “tutorial”. Thanks 🙂

I also would like a PDF copy of this OpenSolaris homer server build if you don’t mind. I find myself referring to this site and using it as documentation quite often.

Hi Simon,

I would like to thank you for making this tutorial. It has been very very good and has helped me many times in my setup of ZFS and Opensolaris.

Cheers!

ps. Can you also send me a PDF of all posts and comments thus far?

Hey Simon,

Thanks for sharing your work and effort to date.
With the info, I’ve been able to build my own – much appreciated.

I am wondering (with the demise of Opensolaris) if you are considering migrating to another platform with a bit more life in it?

I’ve installed 2009.06, which apparently has known issues with CIFS (just dies, requires poweroff/on to fix) and my (personally) NFS isn’t great for any sustained copies over 500MB.
Looking into the CIFS (most used by me) issue, the suggestion is to “upgrade” to the Dev build…

I am considering it (it looks pretty easy).. and or migrating the OS to a freeBSD-based distribution instead.

thanks,
Matt

too bad oracle had to get in on the picture.

ellison = nazi

Hi, your site inspired me to set up my home ZFS server. I have limited experience with OpenSolaris but I have been able to muddle through and it’s worked for many months very well. I ran into a problem where a couple drives were having issues at the same time and I was wondering if you might offer some of your expertise. I was able to put new drives in and get things working except now I have a checksum error in one snapshot file. I have tried to clear the pool and resilver. I have tried to destroy the snapshot which contains the error. I cannot re-silver both the drives without getting this error in this one file. I was wondering if there is a simple way to solve this problem since I really don’t want to get rid of the pool and start over again. I am using snv_134 build of OpenSolaris. Here is the output from Zpool status -v that shows the error. I would like to just get rid of the file and the error with it but I don’t know how to do that in a snapshot.

pool: tank
state: ONLINE
status: One or more devices has experienced an error resulting in data
corruption. Applications may be affected.
action: Restore the file in question if possible. Otherwise restore the
entire pool from backup.
see: http://www.sun.com/msg/ZFS-8000-8A
scrub: resilver in progress for 12h0m, 53.91% done, 10h15m to go
config:

NAME STATE READ WRITE CKSUM
tank ONLINE 0 0 2
raidz2-0 ONLINE 0 0 2
c7t0d0 ONLINE 0 0 0 98.3G resilvered
c7t1d0 ONLINE 0 0 0 98.2G resilvered
c7t2d0 ONLINE 0 0 0
c7t3d0 ONLINE 0 0 0
c9d0 ONLINE 0 0 0
c9d1 ONLINE 0 0 0

errors: Permanent errors have been detected in the following files:

tank/itunes@zfs-auto-snap:daily-2010-10-11-00:00:/iTunes/Album Artwork/Cache/7C7DE397DD748486/07/09/00/7C7DE397DD748486-872B56E2544E1097.itc

That’s kind of what I had suspected was the case. I keep getting more and more errors and I think the pool is beyond hope. Looks like I’m going to have to destroy the pool and start over. I was backing up the essential files on other drives but there was large amounts of data that is not critical but is a pain to replace. This has really shown me the importance of backing up the whole pool.

I bought a 2 TB USB external hard drive and I am going to start doing regular backups of the entire filesystem in case this ever happens again. I guess I got a little overconfident figuring with raidz2 that I would have to lose 3 drives before data corruption.

Thanks!

Simon,

Of all the guides and sites of looked at over the past few weeks, yours appears the most easy to understand:) I have a scenario for you (and anyone else who’d care to reply):

After 16 years of being an IT Engineer in a Microsoft-based environment, I bought an iPhone 3G a couple years ago and have given up on Microsoft for personal use. My home is 90% Apple with a sprinkle of ‘other’ here and there for budgetary purposes. I’ve run out of storage space on a couple of external USB drives and am about to build some type of file storage system. Here is my current setup at home:

Main System – iMac 21.5″, 2.8Ghz C2Duo, 4GB, 500GB.
– Running iTunes, Plex Media Server, Air Video Server

Secondary Systems – Apple TV2, LG Blu-Ray (DLNA), iPhone 4, iPad
– Running Plex Client on all but LG Blu-Ray for media streaming in-house and over internet via wi-fi/3G.

Backup/File Storage – (2) 1TB External USB HDD’s.
– Uses: Time Machine Backup of iTunes, iPhoto, Documents and File Storage for Media Sharing.

Here is what I would like to do:

– Add an 8-10TB File Server/NAS to my network.
– Compatibility with both OS X and Windows for streaming.
– RAID 5 or 6 Redundancy (know RAID5, learning about 6).
– Ability to expand storage later if necessary without complete rebuild.
– Move remaining media off iMac and external drives to run off server, create iTunes Server.
– Use USB External’s for Time Machine backup’s of documents, photos and home movies.

Here is the equipment I’ve chosen based on existing and budget:

– Existing Intel Xeon E3110 Dual Core @ 3.0Ghz, LGA775
– New Gigabyte GA-G31M-ES2L Mobo
– New Crucial 4GB DDR2 800 Memory
– New (2) Promise SATA300 TX4 4-Port PCI SATA II Controller
– New (6) WD Caviar Green 2TB 3.5″ HDD’s
– Existing Full ATX Tower Case with 12 total drive bays
– Existing Antec 500W Phantom PwrSupply

Based on the information above, is your guide up-to-date, any recommendations/changes as well as any additional assistance to a novice Unix/Linux/Solaris user? Oh and I have to stay under an $800 budget with the Tower, PS and CPU already purchased/not included.

Please advise…this looks to be the best option and your guide to be the easiest to follow.

Brad

I just saw tons on warnings on the new EARS, but I will stick with them (only 6 on the other hand).
Problems:

– 4k drives are emulated as 512-bytes sector for s****y Windows XP. Huge and catastrophic impacts on the performance under ZFS because of WD’s care for retarded customers -> I will use the gnop tips to create a virtual disk with 4k sectors as first of my pool + I could also use a modified zpool that lets me specify the ashift from 9 (bad) to 12 (good).
So there is at least two solutions for this issue. But I recognize we should not be constraint on using work-arounds, and WD should definitively release a non emulated firmware to its customers. BTW, tons of new HDDs are also concerned and are using this new method of partitioning, so it is not a Caviar Green only issue.

– TLER setting hardwired on OFF since one year. Apparently, there is no real impact under RAIDZ, in fact, TLER should be DEACTIVATED under a RAIDZ configuration ! so no issue at all, this is just a big confusion around

– Finally, the too many parking issue (load-unload cycles that are raising dramatically). Because it is an energy efficient drive, heads on Caviar Green are parking too often under a RAID configuration, slowing down the whole thing and decreasing dramatically the lifespan of the whole. Lucky we are, there is an utility that is still working on the WD website (WDIDLE3) to set the delay from couple of seconds to 5 minutes, which is way better. Problem solved.

So I will stick with the WD 2 TB EARS. Pity that the only real issue (the 4k advanced format), that is not restricted to the Caviar Green series, is still present and needs a work-around, but I don’t like other offers.

Even though this series is based on OpenSolaris, much applies to FreeBSD as well. I have been using a FreeBSD installation, booting from Compact Flash with a vdev of 3 disks with expansion of a vdev of another 3 disks, for quite a while now, much to my satisfaction

fd0

Thanks David.

Yes, I was also saddened to see what Oracle did with OpenSolaris.
However, illumos appears to be the new code base and, like you pointed out, there are a few derivatives to choose from.

IIRC, SmartOS is developed by Joyent and is optimised for data centres.
OpenIndiana is probably the closest to the original OpenSolaris system and, so for that reason, I chose OpenIndiana as it most matched my requirements for a general purpose drop-in replacement for OpenSolaris.

I upgraded from build 134 of OpenSolaris to build 151a4 of OpenIndiana. There were one or two issues due to a problem in pkg, but thanks to the very helpful #oi-dev guys, I managed to work around the problems at the time. I was thinking of publishing a guide showing how to update the code to OpenIndiana, as it might be of interest to others too.

Hi Simon,

I must tell you that your blog led me down the path of using Solaris as my media server backup machine, and I have been very pleased.

I did upgrade to Oracle Solaris Express 11 before I full realized the consequences. I did not upgrade the ZFS pools beyond what is compatible with OpenIndiana.

I would very much enjoy seeing a write up from you on how you did the upgrade, and if you have any tips on how I may get my machine over there to OpenIndiana, I would be very grateful.

I second David’s suggestion. I have Nexenta in my home server now and would like to upgrade to OpenIndiana maybe, in any case to a distro that is under development…

You sir are the man!