Home Fileserver: RAIDZ expansion
Recently I decided to expand the storage capacity of my ZFS home fileserver’s ZFS storage pool, which was configured to use a single RAIDZ vdev comprising of three 750 GB drives.
I wanted to expand the RAIDZ vdev by adding one extra 750 GB drive.
As you may or may not be aware, it is currently not possible to expand a RAIDZ vdev in ZFS, which is a great pity.
However, it is still possible to achieve this expansion goal, but you have to perform a few tricks. Here is how I did it.
The way to achieve RAIDZ vdev expansion is to destroy the pool and re-create it, using the additional drive(s).
Of course, this means you need to back up all the data first, so you’ll need to have an additional storage location available, which has large enough capacity.
I decided to do two full backups of all my data.
The first backup I did was by attaching one additional SATA drive, typing ‘format’ to see the id of the new drive. Once I had the id of the new drive, I created a second ZFS storage pool that used only this new drive:
# zpool create backuppool c3t0d0
Once the backup pool had been created, it was simply a matter of copying the file systems from the main storage pool to the new backup pool. Of course, if the backup pool required more capacity than a single drive provides, then two drives can be used.
Once the transfer of all the file systems had completed, I made a second full backup by copying the file systems to an iSCSI-connected storage pool on a separate backup server. For setup info see: Home Fileserver: Backups.
After both full backups had been made, I verified that the backups had been successful. All seemed well.
The next step was to destroy the main storage pool comprising of the three-drive RAIDZ1 vdev (single parity RAIDZ vdev, as opposed to RAIDZ2 which uses double parity). This moment is not a pleasant one, but with the backups done, nothing could go wrong, could it?
So the storage pool ‘terminator’ command I executed was:
# zpool destroy tank
Gulp.
The next step was to re-create the new ZFS storage pool using the additional drive, which I had plugged in. First step is to get all the drive ids with which to create the new pool. For this, I used the good old format command:
# format
Searching for disks...done
AVAILABLE DISK SELECTIONS:
0. c0d0
/pci@0,0/pci-ide@4/ide@0/cmdk@0,0
1. c1t0d0
/pci@0,0/pci1043,8239@5/disk@0,0
2. c1t1d0
/pci@0,0/pci1043,8239@5/disk@1,0
3. c2t0d0
/pci@0,0/pci1043,8239@5,1/disk@0,0
4. c2t1d0
/pci@0,0/pci1043,8239@5,1/disk@1,0
Specify disk (enter its number): ^C
#
Then, using the drive ids from above, I created the new storage pool comprising a single RAIDZ1 vdev using the four drives:
# zpool create tank raidz1 c1t0d0 c1t1d0 c2t0d0 c2t1d0
After creating the new storage pool, I took the opportunity of reviewing the file system layout and making a few changes by creating a new file system hierarchy that better suited my needs.
Once I had created the file systems using the new layout, the next step was to copy the file system data back from the backup pool to the new main storage pool.
After I had successfully copied all the data back to its new larger storage pool, I decided to unplug the backup pool drive from the case, and keep it on the shelf in case it was needed again in future. This was done by exporting the backup pool to flush any pending writes before removing the drive from the case:
# zpool export backuppool
So, the end result of this work is that I now have a storage pool using a single RAIDZ1 vdev but utilising four drives instead of three. Using 750GB drives, this means instead of a pool comprising ~1.4 TB for data capacity and ~0.7 TB for parity data, the new setup with 4 drives gives ~2.1 TB for data and ~0.7 TB for parity data. Nice! Grrrrreat.
For more ZFS Home Fileserver articles see here: A Home Fileserver using ZFS. Alternatively, see related articles in the following categories: ZFS, Storage, Fileservers.
Popularity: 33% [?]
Hi Simon,
great blog entry, as always! I like the mix between casual style and solidly founded tech content.
Is there any particular reason why you prefer rsync over zfs send/receive for performing the backups?
Cheers,
Constantin
Hi Constantin, thanks a lot for the compliments and glad you like the writing style!
In the linked-to backups article, I used rsync simply because I was not familiar at that time with zfs send / receive. Later, I wrote about using zfs send / receive for performing backups in this article: Home Fileserver: Backups from ZFS snapshots.
Thanks for writing all the great posts about ZFS on your blog, and I look forward to reading more in future. It was your blog and Tim Foster’s blog that inspired me when I first started looking into ZFS, so thanks again!
Cheers,
Simon
I have really enjoyed reading this series of posts, as I am in the planning stages of my home file server.
Unfortunately, I just can’t justify jumping into ZFS when you cannot expand a raidz vdev. I know drobo and symilar units are expensive and proprietary. But the functionality of being able to expand an array by adding additional disks is just too valuable to me, and I would think it is similar for most other home users.
Call me strange, but I don’t see much point in growing a raidz pool if you need enough temporary storage to store all of your data for backup anyway. At that point you could just use those drives to form a second raidz vdev and add it to your existing pool.
Is there any hope that ZFS will add this functionality? Don’t worry, I’m not holding my breath.
Thanks Eric.
As you say, not being able to expand a RAIDZ vdev needn’t be a problem if you are happy with simply adding an additional RAIDZ vdev when expanding your pool. I wanted to add just one drive though, so adding a new vdev wasn’t an option. The backup storage was no problem as I had sufficient spare capacity available.
I have noted my comments on Drobo here: http://breden.org.uk/2008/03/02/home-fileserver-existing-products/
The points I mentioned about Drobo were:
Yes, I think there will be a solution available for expanding RAIDZ vdevs sometime. The last time I looked, there was some info on how this might work here: http://blogs.sun.com/ahl/entry/expand_o_matic_raid_z
ERIC,
You can not easily expand an existing vdev as of now, but you can create a new vdev and add it to the ZFS raid.
Say you have 3 drives in a ZFS raid. Then you can add 3 more drives to that raid. So now you got a ZFS raid with 3 drives + 3 drives. On each set, one disc will be used for parity, so here you have 4 drives for storing data, and 2 drives for parity. All data will be shared on the ZFS raid, spread out evenly on the discs.
Alternatively, you could have created one ZFS raid with 6 drives. Then one of the drives would have been for parity and store data on 5 drives. This is the approach used in this article.
(When using many drives, 10 or more, the prefered approach is to have several smaller sets combined into one large ZFS raid. This is prefered on the SUN thumper machine which has 48 drives. Many smaller sets combined into one big ZFS raid is recommended. Just buy several SATA I/O cards with 8 connections, and you can have a thumper at home. Remember, ZFS prefers no hardware raid card. ZFS likes to have control of everything by itself. No hardware raid cards then).
Of course, you could add only one drive to your first set of 3 drives. Then you would have a ZFS raid with 3 drives + 1 drive. This would be bad, because what happens if that 1 drive fails? In the set of 3 drives, one drive can fail because that set has redundancy. But the second set has no redundancy. Therefore, add a group of drives to an existing ZFS raid. Within a group, the drives should have equal storing capacity.