A Home Fileserver using ZFS
For many people who use a computer, knowing where to store growing amounts of data can become tricky.
You start off with one disk, run out of space, buy a bigger one etc. And if you have a camcorder you’ll be generating gigabytes of data for every Mini DV tape you record. Also, you may have a digital video recorder attached to your TV and wish to permanently keep some of the programmes/films you’ve recorded. Now you’re talking hundreds of gigabytes, if not terabytes of storage that are required to handle all this data.
And then there’s the problem of backups… oh boy, this will be a fun project 
Here’s a series of articles that tackle this tricky subject, where I describe the choices I made, the problems encountered and the solutions found during my quest to build my own ZFS home fileserver, or ZFS home NAS box (network attached storage).
- Home Fileserver: What do I need?
- Home Fileserver: Existing products
- Home Fileserver: I’ll use ZFS
- Home Fileserver: ZFS hardware
- Home Fileserver: ZFS setup
- Home Fileserver: Backups
- Home Fileserver: Suspend
- Home Fileserver: Trunking
- Home Fileserver: ZFS snapshots
- Home Fileserver: Backups from ZFS snapshots
- Home Fileserver: Drive temps
- Home Fileserver: RAIDZ expansion
- Home Fileserver: Active Directory Integration
A quote from Paul Venezia, InfoWorld: (click the picture to go to the full InfoWorld article)
It’s not every day that the computer industry delivers the level of innovation found in Sun’s ZFS. The fluidity, the malleability, and the scalability of ZFS far surpass any file system available now on any platform. More and more advances in the science of IT are based on simply multiplying the status quo. ZFS breaks all the rules here, and it arrives in an amazingly well-thought-out and nicely implemented solution.
We’re talking about a file system that can address 256 quadrillion zettabytes of storage, and that can handle a maximum file size of 16 exabytes. For reference, a zettabyte is equal to one billion terabytes. In order to bend your mind around what ZFS is and what it can do, you need to toss out just about everything you know about file systems and start over.
For more information on related subjects, see articles in the following categories: ZFS, Storage, Fileservers.
Popularity: 100% [?]
Hi,
very interesting project - can you tell me which version of solaris did you use ? there seems to be Solaris 10, Open Solaris, Solaris Developers Edition …. Which one is the most suitable for a stable, reliable fileserver ? I’m after data integrity (zfs), redundancy (raidz) and something simple to admin that “just works!”
thanks
CW
@Chn Wng: I used Solaris Express Community Edition: get it here:
http://opensolaris.org/os/downloads/sol_ex_dvd_1/
Thank you for this. Your walkthrough persuaded me to give ZFS a try over unRAID and (heaven forbid) WHS.
One of the few things that originally had me leaning towards a WHS machine was the availability of a simple plug-in for implementing off-site backups using Amazon’s S3 service.
I found this and plan to give it a go:
http://developers.sun.com/solaris/articles/storage_utils.html
Have you tried something similar? I appreciate the information on incremental snapshots.
ZFS seems to be far and away the most powerful means for implementing a home NAS.
Hi Sal, thanks for the feedback — I’m a sucker for compliments
Amazon’s S3 could be interesting. Do you know the current prices they charge for storage? I didn’t try this yet.
Yes, the snapshots for incremental backups seem a great idea.
And I agree, ZFS does seem to be the most powerful thing right now for a NAS, and I don’t see anything replacing it any time soon.
Good luck with ZFS!
From what I’ve seen of people using Amazon S3 for off-site Windows Home Server backups, they pay only a few US Dollars a month for the service.
This page has the info:
http://www.amazon.com/gp/browse.html?node=16427261
They charge for the amount stored and the amount transfered. I’d have to imagine that snapshots are generally very small for a system used this way (very few deletions and/or changes) so the transfer costs would be minimal.
I don’t know how Amazon S3 compares to other services, but it seems to be the go-to service for WHS off-site backups. Good luck!
Thanks Sal, I’ll check that link out.
I’m surprised no one considers either NexentaOS or NexentaStor, a prebuild NAS using ZFS.
Hi Joe, I’m aware of NexentaOS/NexentaStor but not used it so far. I’ve heard some good things about it, so it might be good to take a look at it one day.
However, for this project, I wanted to try and use a standard OpenSolaris installation and build something completely from scratch, so that I could learn and understand how it all works.
Also, I didn’t want to use a free version of a commercial product having a 2TB size limitation, if I recall correctly, in case I needed more space one day.
Hey Simon, I’m still playing with ZFS on vmware while my hardware is on order. I like to familiarise myself with it as much as possible. There’s just one thing that I don’t seem to be able to find an answer to anyway and I wonder if you can help.
I’ve got a zfs shared with smb, not samba as i understand it but CIFS. SMB authentication through pam is working fine and I can create files on the share perfectly. The only problem is that after a file is created, it is assigned no permissions whatsoever. I know that using samba I would use the “create mask” option, but seeing as this is CIFS I’m a little stuck. Google hasn’t brought me much success and the osol-discuss mailing list hasn’t thrown any answers either.
Thanks
Hi Matt, I think what you’re looking for is ACL and there is a complete chapter about how it works in the ZFS Administration Guide: http://opensolaris.org/os/community/zfs/docs/zfsadmin.pdf — see the chapter entitled ‘Using ACLs to Protect ZFS Files’. Hope this helps.
I am going to try setting this up but don’t have much experience. My question is… can I use a bunch of different size drives for ZFS or do they all have to be the same. I will be using this to serve media throughout the home as well as store work docs so that I can access from any computer. Ie. a pool for each of these: movies, music, pictures, home videos, work
My primary goal is to be able to serve these up quickly over my gig lan but I also don’t want to lose the data. I have 3 750gb disks and then I have 4 that range from 160 to 300. Would I just setup two different sets of pools? Ie. use the 3 750gb just like you set it up and then create pool using the 4 disks. sorry if this is really basic but I just don’t understand as I am very new to this.
Thanks
Simon, very nice writeup! I’ve just finished a similar project for my home file server. I wish that smartmontools would support SATA drives so that I can monitor HDD temps as you do.
http://home.comcast.net/~robhensel/NAS/Solaris_10_NAS.htm
Hi JV,
the short answer is it depends what kind of pool configuration you want. If you want to create a simple RAIDZ1 configuration then it’s best to have the same size drives, probably the same make and model too — i.e. I create a pool comprised of a single RAIDZ1 vdev that uses three 750 GB drives.
Or, if you had four drives, say two 500 GB drives and two 750 GB drives then you could create a pool composed of two mirrors, one mirror using the two 500 GB drives, and the other mirror composed of two 750 GB drives.
The problem with trying to make vdevs from multiple different sized drives is that each disk will have the capacity of the smallest drive. For example, if you had three drives — a 160 GB, a 320 GB and a 750 GB drive, and you formed a RAIDZ1 vdev from these, ZFS will make all the drives 160 GB in size, so you’d only be able to create a vdev from these which has a capacity of 480 GB (3 x 160 GB).
If you have a bunch of odd-sized drives, then perhaps you can use them to make a backup pool, and use no redundancy, thereby getting the full capacity of each drive. This is exactly what I did to make use of a bunch of old SATA drives I had — and I made a backup server from all these old drives.
Hopefully that answers your question.
Sorry, been a long day, and re-reading your question I might make the following suggestions:
1. Make one big pool using the three 750 GB drives, probably best to use a single RAIDZ1 vdev — i.e. zpool create mypool RAIDZ1 disk1 disk2 disk3
2. Once the pool is created, I would create 5 file systems within this pool for each of: movies, music, pictures, home videos, work. You can then assign properties to each file system independently. For example, you might decide to have only one copy of each file within the movies and music file systems, but you might decide to create 2 copies of each file within the pictures, home videos and work file systems, as they are content created by you which cannot be recovered if data is lost. For this see the ’set copies=2′ attribute for the file system. Some people prefer to create 2 disk mirrors for personally created content, as each disk in the mirror is an exact copy of the other one.
3. See my comment above regarding using 4 different sized disks. Basically, you have 2 choices if you include a vdev in your pool comprising these 4 different-sized disks: (1) create a RAIDZ1 vdev but then ZFS will use the smallest disk size for all 4, or (2) create a vdev with no redundancy and be able to have the full capacity of all the drives added together — but the problem is that if there’s a an error you will probably lose the affected data permanently! For this reason, like I did, it may be best to do this (no redundancy) only for a backup server/pool, where you have the original data in the main server’s pool. It’s your choice.
Hi Rob,
Thanks! smartmontools does support SATA drives, but for some reason, the notation specifies SCSI. I know it works because I am using smartmontools with my SATA drives and the script here: http://breden.org.uk/2008/05/16/home-fileserver-drive-temps/
I’ll take a look at your writeup once I’ve finished replying to this stack of comments… (returned from holiday)
Hi Simon,
you posted a problem concerning using “cp” to copy files at http://www.opensolaris.org/jive/thread.jspa?threadID=59201
Are there any new information about that problem? Did you try the latest snv_91?
Best regards
Chris
Simon, as I’ve said previously I’ve been following your setup almost exactly and so far I’m extremely happy with how things are going. The main way my deployment is differing from yours is that I’m using a windows 2003 Active Directory domain and i’ve linked my solaris box into that, using centralised permissions etc.
I’ve got some information on my experiences with CIFS and Active Directory which you’re welcome to post here to complete your excellent docs on the subject
Hi Chris, no I have no more news on that bug as I’m still on snv_87. However it looks like bug 6669134 was fixed in snv_90, so it’s possible that the bug has been fixed now. However, there were a number of similar-looking bugs that I listed in that post, and some of them have not been fixed yet, so it’s quite possible that the bug still remains in snv_91.
If you try snv_91, it would be great if you can confirm whether you see the bug. Likewise, when I upgrade, I will note here if I find the bug has been fixed.
Cheers,
Simon
Hi Matt, glad you’ve got it working nicely. And it’s good you’ve managed to make it work with Windows 2003 Active Directory Domain.
Personally, I try to avoid using Microsoft products as much as possible, so this may not be useful for my setup, but I could link to your content if you’d like, once you’ve got it ready.
Great write up.I appreciate it.I’ve been planning a new NAS set up and this really looks good.
Al
Thanks a lot Al ! Believe me, probably like yourself, I spent a lot of time hunting down the best storage solution, and ZFS is it. Enjoy!
Hi Simon,
I made an upgrade on my OpenSolaris 2008.05 machine. The kernel is now snv_91 and the problem is still not solved. Too bad!
Best Regards,
Chris
Hi Chris, thanks a lot for your feedback regarding this bug. That’s a real pity that they haven’t fixed it yet. I have been too busy to learn how to use DTrace to debug this problem as the code in the nv_sata() driver is called by the ‘cp’ command for this bug.
If you have the time and the knowledge to look at the possibility to run DTrace to see if anything can be discovered about this bug, see the last post from ‘bhorn’ here: http://www.opensolaris.org/jive/thread.jspa?threadID=59201#233503
Otherwise, I will try to debug it, but it won’t be very soon, as I have a whole load of stuff to keep me busy right now. Keep in touch!
Cheers,
Simon
I openend a bug. http://defect.opensolaris.org/bz/show_bug.cgi?id=2366
I already got a hint how to debug this behaviour. I hope to get it done later today.
Chris
Good news Chris! I look forward to seeing where it hangs.
Simon
My question is are you using a full solaris install or something that only includes the ZFS, NFS and CFIS functionality maybe some iSCSI? It seems like all the home file servers are using the bloat approach, I’m curious if anyone has gotten it down in the 128MB ~ 64MB range for a true appliance style.
Hi Simon,
it’s so weird. Since a few days the bug does not appear anymore. We tried to reproduce the bug for the bugreport, but it just works. 5GB files. 16GB files. No problem at all. But I’m distrustful. Bugs like this one don’t disappear like nothing!
Would be great if you gib snv_91 a try and maybe report your experience with the release. Maybe it’s fixed already.
Chris
Hi Dave, I’m using a full, standard OpenSolaris install — Nevada, build 87 currently. I believe Nexenta may be what you are looking for, but be aware that you only get a free version capable of running a 2TB array, last time I looked. If you will ever need to store more than 2TB then you will be paying to use this software. That’s one of the main reasons I chose to use the standard version of OpenSolaris.
Hi Chris, that’s very strange, but I can bellieve it. I had the same thing happening when I was trying to reproduce the bug, and then the bug would reappear. This is not the kind of bug that you want to have in a system that is meant to be virtually bullet-proof. When I get some time I will reinstall with the latest Nevada install (b93 currently) and give it another try. I don’t think it is fixed in snv_91 as I seem to recall that I reproduced the bug with that version, and then went back to snv_87, but found the bug also in snv_87. I will let you know what I discover in the coming weeks after I upgrade.
Simon
Okay Simon. Looking forward to hear from you.
A note on snv_93: I barely use it, but the ZFS web administration interface seems to be broken (again!). But ZFS boot in the installer is very nice!
Have a nice weekend!
Take care,
Chris
Hi Chris. Likewise, I don’t use the ZFS web admin interface — I just use the command line and scripts. I will be expanding the array this weekend, and if I get time after that, I might install snv_93. Have a great weekend too!
Cheers,
Simon
Simon on June 26th, 2008 at 18:40 wrote:
> Hi JV,
> the short answer is it depends what kind of pool configuration you want. If you want
> to create a simple RAIDZ1 configuration then it’s best to have the same size drives,
> probably the same make and model too — i.e. I create a pool comprised of a single
> RAIDZ1 vdev that uses three 750 GB drives.
True.
> Or, if you had four drives, say two 500 GB drives and two 750 GB drives then you could
> create a pool composed of two mirrors, one mirror using the two 500 GB drives, and the
> other mirror composed of two 750 GB drives.
Using OpenSolaris / Solaris / SXCE / etc. you could partition the drives into 250GB chunks
and then RAID-Z2 / Mirror the pieces back together. If you do it correctly then the different
chunks on different drives / controllers will each protect one and other.
Depending on your configuration you can add the 500GB drives to the 750GB drives and then
mirror / raid them and end up with 1250GB of drive space with twice the speed (IOPS) or
take the total of (2 * (500GB + 750GB)) 2500GB and use a RAID-Z2 9+2 to get a huge MTTDL
and a lot of drive space.
Lots of ways to slice it
. Partitioning improves defrag times, it is not evil.
Joel
Hi Joel,
Although zfs pool creation using slices is possible to do, it appears that it’s not regarded as a best practice, from what I have seen. The best practice seems to be to use complete drives. See: http://www.solarisinternals.com/wiki/index.php/ZFS_Best_Practices_Guide#Storage_Pools
There it says:
Set up one storage pool using whole disks per system, if possible.
For production systems, consider using whole disks for storage pools rather than slices for the following reasons:
* The recovery process of replacing a failed disk is more complex when disks contain both ZFS and UFS file systems on slices.
* In general, maintaining slices increases administration time and cost. Lower your administration costs by simplifying your storage pool configuration model.
A pool created with multiple slices across disks is harder to manage than a pool created with whole disks.
Hi Simon,
if you use CIFS as a service don’t update to snv_93!
http://www.opensolaris.org/jive/thread.jspa?threadID=65996
I haven’t tested CIFS that much, but it seems to be a big problem.
So if you not already updated to snv_93, wait for snv_94!
Regards,
Chris
Hi Chris, thanks a lot for the warning about snv_93 — I will not try it
Cheers,
Simon
snv_94 was released today! Hopefully the bug is fixed!
I will install snv_94 on my filer, too.
Best regards,
Chris
Thanks Chris, I hope the bug has gone this time — let’s hope.
The bug is gone.
http://www.opensolaris.org/jive/thread.jspa?threadID=65996
Let’s try snv_94!
Regards,
Chris
I’ve been reading this and it’s one of the reasons i’ve chosen to build a computer from scratch and use opensolaris for its ZFS capabilities as my file/backup server. Does anyone know if the web utility for configuring ZFS is working or not? I saw it in a write-up about opensolaris developer express edition, but i think now that 2008.05 is out, it is not available. Are there any good books out pertaining to ZFS or opensolaris?
thanks,
brian
Thanks Brian. Not sure about the web interface. Honestly, ZFS is so easy to setup with the command line, that I didn’t consider the web interface necessary. Good luck with your new system!
Hi Simon. As we spoke about a while ago, I have written a small guide to integrating your Solaris ZFS/CIFS fileserver with Active Directory for centralised permissions. I would be happy for you to link it somewhere in your guide in case others are interested in this combination.
It is the first document on my new wiki, you can see it at http://www.genestate.com/OpenSolaris:ActiveDirectory_Integration
I would be happy to recieve any feedback people have on the guide and I will be expanding it as soon as I have some free time. It’s not really a unique guide, but it is a compilation of all the things I have found while working it out for myself
Hi Matt,
Fantastic news. If you could send me some summary blog text, I will use it to create a new post, and will link to your article from there to give the full details that you have written.
Thanks a lot.
Simon
>I believe Nexenta may be what you are looking for, but be aware that you only get a free version capable of running a 2TB array, last time I looked. <
Actually, you should be aware that the 2TB limitation is only on NexentaStor. NexentaOS and NexentaCore have no limitation on the size of any pool - they are completely open source, built on Debian and OpenSolaris. Get them from nexenta.org. The administration and reporting tools they’ve added to NexentaStor are closed source, and that’s where the 2TB limitation comes. If you already use just the command line and scripts, there would be no penalty to using NexentaOS or NexentaCore. (NexentaOS is the desktop version, NexentaCore the basic server version.) It’s supposed to run on lower-spec and a broader range of hardware.
Hi Bryan,
In fact it’s worse now for NexentaStor: they’ve reduced the limit to 1TB now for the Developer Edition, or instead you can choose the Free Trial Edition with no capacity limits stated but you have a 30 day time-limit. As the scope of this post is for a home fileserver/NAS, NexentaStor is not suitable, as the annual ongoing costs are prohibitive — it’s meant for businesses.
Personally, I had no use for NexentaOS/NexentaCore as all that I needed was already in OpenSolaris, which is also free. Also, the APT features have now been included in OpenSolaris 2008.05, known as IPS (Image Packaging System), which you can use to download and install free security fixes. However, I believe that you have to pay for the ability to have full use of IPS for updating other packages (http://www.sun.com/service/opensolaris/index.jsp), and this appears to be an ongoing $324 payment. Again, as the scope of this post is for a home fileserver/NAS, this is unacceptable. For businesses, that may be another matter.
However, if you’re saying that NexentaCore/NexentaOS are free, have no restrictions of any sort, and allow easy updating via APT then it could be interesting.
But I do like the fact that by using the basic vanilla OpenSolaris, I know exactly what I’m using (which version etc), and can fire off questions to the great enthusiasts in Sun’s OpenSolaris ZFS forums, for example, or general driver questions, and there’s fewer external things to depend on. Anyway, for what I’ve needed here, this approach has worked fine so far.
Simon