The configuration I would strongly recommend you use is the following, comprised of two pools:

1. ‘rpool’ : one pool containing / and /boot. This contains the boot environment and root file system. Mirror this using two drives to give protection. You’ll also benefit from GRUB bootable ‘Boot Environments’ allowing you to recover from failed OS upgrades (like Debian’s apt-get -u dist-upgrade). This has saved my boot system on one occasion already.

2. ‘tank’ : one pool containing all your non-OS/boot data. Put all your user data here. This corresponds to your previous ‘/srv’

‘rpool’ can be mirrored using a ZFS ‘mirror’ vdev.

‘tank’ can be configured to use any single vdev or combination of vdevs you like. My recommendation, assuming you have the cash to invest, is to create a single RAID-Z2 vdev for this, for simplicity of administration and strong protection of data. As the capacity of two drives is used for parity data with RAID-Z2 vdevs, you will probably want something like capacity of 4 drives for data and capacity of 2 drives for parity — i.e. 6 drives in total for user data.

However, as you only have 6 SATA ports on your mobo, and you will need to reserve 2 for your root/boot pool, you might have to reconsider this and either economise to a 4-drive RAID-Z1 vdev for user data, or upgrade to using a PCIe SATA HBA like an AOC-USAS-L8i or an LSI LSISAS3081E-R, both of which give you a further 8 SATA ports to use as you choose.

I am using the AOC-USAS-L8i, after having gone through two upgrade processes. If cash is not a problem for you then I would recommend investing in one of these cards to give a strongly protected boot/OS + data storage environment.

Cheers,
Simon

I’m pretty new to OpenSolaris. I had a Debian NAS but the performance of Samba was not really good, so I tried other systems. OpenSolaris is really fast. But I’d like to have a system like my old Debian installation. I have a mainboard with 6x S-ATA connectors and I have six S-ATA hard disks. On Debian I had the following setup, all hard disks had the same partitioning:
1GB /boot, RAID1 using all hard disks
5GB / (root), RAID5 using all hard disks (25GB for root)
about 700GB /srv, RAID5 using all hard disks (the RAID for CIFS / Samba)

Is it possible using ZFS to create two or three pools on the hard disks so I can install OpenSolaris like my old Debian installation?

At the moment I’m trying OS on my USB stick, but this is a slow solution and seems not to work pretty good…

Regards, Nils.

You’re right, you do need a lot of space available. A backup system should help with this, as all your data should be backed-up, difficult as it can be.

RAID is not backup, and this is often forgotten by people thinking of implementing a RAID setup.

vdev expansion is not a priority for Sun (Oracle soon?), as enterprise users simply buy large arrays to start with, and then upgrade them by adding another 6 or 12 drives etc, as a new vdev. Adding vdevs to an existing pool is possible, but not expanding an existing vdev.

But you’re right, having the ability to expand a vdev would be a useful feature for home users. But Sun won’t get any money from us home users.

There is some recent work by Matt Ahrens which may prepare the way for vdev expansion one day… soon?
He says: ‘This work lays a bunch of infrastructure that will be used by the upcoming device removal feature. Stay tuned!’
See: http://blogs.sun.com/ahrens/entry/new_scrub_code
It might look unrelated to vdev expansion, but if he says the on-disk changes should allow upcoming device removal, then it seems likely that it might also facilitate device addition.

So you have to waste a lot of space.
It’s a shame, that RAIDZ still can’t be expanded!

Yes, I knew that method was a possibility, but it would have required a full resilver after each of the three disks was replaced, which (a) would have taken a long time, and (b) could have toasted my data if an existing drive had failed during any of the three required resilver operations.

There are other issues with the ‘replace one disk at a time’ method, and that is:
1. I would have needed to buy three more larger disks (e.g. 1TB) to replace the existing operational 750GB drives.
2. There is no backup available in the event of the operation failing.
3. I would have three 750GB drives with no use at the end of replacing them all with 1TB drives (I prefer to have one vdev currently, due to the limitation of existing available SATA connectors and drive cage space).
4. I would have only gained ~750GB by replacing the three existing 750GB drives with 1TB replacements, which is the same that I gained by recreating the array with one additional 750GB drive.

Hopefully this makes sense

Cheers,
Simon

You can not easily expand an existing vdev as of now, but you can create a new vdev and add it to the ZFS raid.

Say you have 3 drives in a ZFS raid. Then you can add 3 more drives to that raid. So now you got a ZFS raid with 3 drives + 3 drives. On each set, one disc will be used for parity, so here you have 4 drives for storing data, and 2 drives for parity. All data will be shared on the ZFS raid, spread out evenly on the discs.

Alternatively, you could have created one ZFS raid with 6 drives. Then one of the drives would have been for parity and store data on 5 drives. This is the approach used in this article.

(When using many drives, 10 or more, the prefered approach is to have several smaller sets combined into one large ZFS raid. This is prefered on the SUN thumper machine which has 48 drives. Many smaller sets combined into one big ZFS raid is recommended. Just buy several SATA I/O cards with 8 connections, and you can have a thumper at home. Remember, ZFS prefers no hardware raid card. ZFS likes to have control of everything by itself. No hardware raid cards then).

Of course, you could add only one drive to your first set of 3 drives. Then you would have a ZFS raid with 3 drives + 1 drive. This would be bad, because what happens if that 1 drive fails? In the set of 3 drives, one drive can fail because that set has redundancy. But the second set has no redundancy. Therefore, add a group of drives to an existing ZFS raid. Within a group, the drives should have equal storing capacity.

As you say, not being able to expand a RAIDZ vdev needn’t be a problem if you are happy with simply adding an additional RAIDZ vdev when expanding your pool. I wanted to add just one drive though, so adding a new vdev wasn’t an option. The backup storage was no problem as I had sufficient spare capacity available.

I have noted my comments on Drobo here: http://breden.org.uk/2008/03/02/home-fileserver-existing-products/

The points I mentioned about Drobo were:

Drobo initially looked quite interesting, especially in its flexibility for handling different sized drives. However, it seems to suffer from pitifully slow transfer speeds due to using a USB2 interface, and its data format seems undocumented and, therefore, proprietary. I didn’t need to look any further. Also the price was around $500 without drives, so it’s not cheap either. I believe there is some kind of bolt-on box called DroboShare for giving ethernet access too, available for another $200. So this gizmo with ethernet access will cost $700 without any drives. No thanks.

Yes, I think there will be a solution available for expanding RAIDZ vdevs sometime. The last time I looked, there was some info on how this might work here: http://blogs.sun.com/ahl/entry/expand_o_matic_raid_z

Unfortunately, I just can’t justify jumping into ZFS when you cannot expand a raidz vdev. I know drobo and symilar units are expensive and proprietary. But the functionality of being able to expand an array by adding additional disks is just too valuable to me, and I would think it is similar for most other home users.

Call me strange, but I don’t see much point in growing a raidz pool if you need enough temporary storage to store all of your data for backup anyway. At that point you could just use those drives to form a second raidz vdev and add it to your existing pool.

Is there any hope that ZFS will add this functionality? Don’t worry, I’m not holding my breath.

In the linked-to backups article, I used rsync simply because I was not familiar at that time with zfs send / receive. Later, I wrote about using zfs send / receive for performing backups in this article: Home Fileserver: Backups from ZFS snapshots.

Thanks for writing all the great posts about ZFS on your blog, and I look forward to reading more in future. It was your blog and Tim Foster’s blog that inspired me when I first started looking into ZFS, so thanks again!

Cheers,
Simon

great blog entry, as always! I like the mix between casual style and solidly founded tech content.
Is there any particular reason why you prefer rsync over zfs send/receive for performing the backups?

Cheers,
Constantin