I would like to thank you for making this tutorial. It has been very very good and has helped me many times in my setup of ZFS and Opensolaris.

Cheers!

ps. Can you also send me a PDF of all posts and comments thus far?

Thanks for the request.

What would you like to see in the PDF? All of the posts, with comments too?

Cheers,
Simon

thanks for your information on OpenSolaris and your Home Server Setup and also for answering all the questions from the comments. So I hope you can also answer my little ones.

I can think of a scenario with one system drive and 4 data drives bound to a raidz1. Now my two questions:

a) How should one backup the system drive? Doing snapshots to the raidz1 pool and in case something bad happens with the motherboard or the system drive put everything back to a new system drive? How would this scenario work in real life?
Is it possible to snapshot the whole system drive and than to clone it back to another disk so that it is ready to boot? What needs to be done to achieve this?

b) Think of one part of the raidz1 pool is dedicated as time machine backup place for a mac and thus exported via iSCSI as a block device to the Mac system. Also deduplication is switched on. What needs to be done to recover the Mac if its disk fails. How do I get the backup back on the new Mac disk. I think I can’t clone it directly back to the new Mac disk?

Thanks for your help and keep up the good work,

Jan.

If you’re more comfortable with stable, infrequent releases, try Solaris 10:
http://www.sun.com/software/solaris/get.jsp
http://www.sun.com/software/solaris/

Choosing which version of Solaris to use is always a tradeoff: (1) OpenSolaris=frequent releases containing new features under development or (2) Solaris=infrequent releases, but heavily tested and rock solid.

Cheers,
Simon

Simon, this unfortunately was very bad advice and I lost several days by installing and wrestling with Open Solaris 2009.06 for use as a file server.

Both smb/server and network/samba are _badly_ broken and as I later learned the hard way Sun admins are fully aware of the situation but did not publish any warnings on the download page.

With all the problem reports google brought up imho Open Solaris is the testbed for Sun and success with such distributions heviliy depending on luck and local weather conditions.

My conclusion: Unless you are prepared for a long and ugly struggle: Avoid Open Solaris!

someone

Second i would like to configure an ftpserver so i can access and transfer files from outside my home network. I really havent found a good guide to do this….
The nas is connected to a router with the firewall enabled, is this enough?
Should i configure the “ipfilter” in opensolaris too? Is there a guide somewhere for this?
I have tried to look at man pages but i havent found anything useful…
As you can see im a unix beginner. I think Opensolaris is a great OS, but its not very userfriendly and
information on how to set up certain bits is not really abundant.
Maybe im looking in the wrong places?

Im now configuring/administrating/checking health on this machine via PuTTY SSH (which is pretty slow)
and sometimes via VNC
Isnt there a faster and easier way to do this??
What web interface alternatives are there?
I have found something called Webmin, is this the best one with the most features that you may need?

Your blog has helped me a lot already, but maybe you can point me in the right direction regarding my
questions?

Thanks a lot!!
Gurkman

If you have basic Linux knowledge then you should find it relatively easy to use OpenSolaris too. You can find the answer to any admin problem online: Google, OpenSolaris forums and blogs etc.

For using Active Directory and Solaris, see the AD link at the top of this page.

Regarding bottlenecks, the network will be the bottleneck when transferring big files, but even with commodity Gigabit ethernet you should get speeds of around 50 MBytes/sec, which should be sufficient. If you need serious speed, look at 10GbE or Direct Attached Storage (DAS) using tech like InfiniBand, which is used with an HBA, and is often used by video editing software to connect to moderately fast storage devices like this, which easily reach speeds of around 500+ MBytes/sec sustained using 8 drives in a redundant array.

Regarding running 2 boxes at home, you can probably ditch Windows and use some Solaris directory service, if you need it, but I didn’t look at this.

Good luck,
Simon

I have a Home Server with Windows 2003 Server (Domain Controller) with Hardware-RAID5. There are about 4 Windows Clients connected to this server.
Primary use of this server is file storage (huge collection of DVDs, images, mp3 collection, pdf scans and documents), some applications and web server.

I’m a windows guy and have only minor knowledge in linux/OS, but I REALLY WANT the superior ZFS.

Is it possible to run OpenSolaris with ZFS and within this system a virtualized Windows 2008 Server (with Active Directory)? I’d like to setup OS and ZFS as a (hopefully) stable “install-and-forget” system. Windows Server would be see the ZFS-Pool as a huge virtual drive and shares this drive to its Windows clients. So all my Windows clients would be connected to the Windows Server only. They (and Windows Server) don’t know of any OpenSolaris Server.
With this scenario I could continue to work with Windows Server, but can trust all my datas to the ZFS.

Is this possible? If yes, is there any performance bottleneck when transfering huge files from the clients to the server (or vice versa)?

Most time, my home server is idle and not very busy. (2 concurrent users)

For my home environment, I don’t want to run two dedicated server (OS for storage and Win2008 for Domain), so I really would like to virtualize it.

What’s your opinion?

And thanks for your great site. Lots of very useful information!

But atm Iam ordering 2 x LSI http://www.lsi.com/obsolete/megaraid_sas_8308elp.html and 2 x Compucase 5in3 Hotswap bays. Then I will have 22 sata ports in total which should last me a good while as I wont buy drives for all this. Will probobly buy 3-5 drives now and then upgrade on the way. I just found a good deal on that hardware.

But I am going to use ZFS, just need to buy big chunks of drives then (4-5) when I cant just add a single drive at a time.

As I havent used solaris before I am tempted to try to just use FreeNas as that also supports ZFS now, just not sure these raid controllers support FreeBSD but they have support for solaris.

You don’t need RAID cards for ZFS, quite the opposite. ZFS controls everything, and so you should use a JBOD setup. But you might want to buy a SATA controller to control your array of cheap SATA drives.

If I were you, I would assume that RAID-Z expandability is not high on the ZFS developers’ priority list, as they have stated this openly, and so I would just assume that you need to create an array with sufficient capacity for your foreseeable needs.

Personally I would not trust RAID level 5 for anything. AFAIK, RAID 5 does not use block checksums and so it has no way of knowing if there is ‘bit rot’ when it reads data back. RAID level 5 writes parity data to allow failed drives to be rebuilt, but if it can’t tell you that some bits in a file have been flipped then what good is it to anyone? Check out various CERN and Google reports on parity errors if you need convincing further.

ZFS was designed with the idea that you should never trust drive hardware, and so that’s why it has end-to-end data integrity built in from scratch, to help (1) be sure that what gets written and read back is actually what you told it to write, and (2) immediate detection of ‘bit rot’ on reading back of a file or during regular weekly/monthly ’scrub’ operations where ZFS will read back ALL the blocks of every file in the storage pool and will check them against their block checksums, and if a difference is detected, ZFS will fix the corruption immediately from available parity data, assuming the vdev containing the corrupted file has redundancy — i.e. mirror, RAID-Z1 or RAID-Z2.

Hope this helps.

Btw would you say going raid5 would be a bad idear compared to ZFS when I dont have 24/7 rated hard drives?

This sounds like a good idea for a new blog post — thanks for the idea
Quick rundown: with ZFS you have the concept of a storage pool. The pool is comprised of one or more vdevs. A vdev is a collection of drives in a particular configuration — e.g. mirror, RAID-Z1 or RAID-Z2 etc. Currently a vdev of type RAID-Z1 / RAID-Z2 cannot be expanded, but Sun say they plan this functionality for the future (see http://blogs.sun.com/ahl/entry/expand_o_matic_raid_z). However, you can expand your pool as much as you like, by simply adding new vdev(s) when you need to, without any problem.

Knowing the above, for now, it’s best to decide how much space you’ll need and build in plenty of storage up-front. Just as an example, it is possible to break a 3-drive RAID-Z1 and recreate a 4-drive RAID-Z1, for example, as you can see here:
http://breden.org.uk/2008/09/01/home-fileserver-raidz-expansion/

Cheers,
Simon

I want to be able to add many more drives to my system, and I think I read in this article that you can not add a single drive at a time, but how many do you have to add then and will you utilize 100% of their discspace or?

I am also trying to figure out if I should run Freenas or Solaris

When you say ‘at first the driver crashed all the time’, was this recently or a long time ago, as I believe there were many problems with the standard ‘SAT’ card (i.e. AOC-SAT-MV8), but that many of these problems were fixed in the later ‘SAT2′ card. I saw that info here, and he also says that the AOC-USAS-L8i (LSI megaraid chips?) would be his next choice — see here:
http://zpool.org/2008/12/16/my-zfs-media-server

He says:

This card [AOC-SAT-MV8] has had a lot of problems with ZFS and has been discontinued by Supermicro in favor of the AOC-SAT2-MV8. Whenever I expand beyond six disks, I’m going to skip straight to the AOC-USAS-L8i using mini-SAS to SATA cables.

I will take a look at your Automounter notes next. If I remember, the reason I used CIFS sharing instead of NFS was because of lousy write speeds (5MB/s with NFS instead of about 40MB/s with CIFS (50MB/s now with OpenSolaris 2009.06 and OS X 10.5.7 using 1 x GbE)), but maybe this has been fixed now. I think it was because of NFS using synchronous writes if I remember correctly. These quoted speeds were achieved writing from a Mac Pro to the ZFS file server.

Thanks a lot for the info. I have created a new thread on the Solaris forum with more questions/info — see here:
http://www.opensolaris.org/jive/thread.jspa?threadID=106210&tstart=0

http://www.provantage.com/lsi-logic-lsi00165~7LSIG06Q.htm

but most freshly-bought systems will find a PCIe motherboard cheaper, in which case you can use AOC-USAS-L8i, an 8-lane PCIe card. though you’ll have to remove the bracket because supermicro’s put it on backwards.

I think you can also use Dell PERC cards with the mega_sas driver, which unlike the other two cards above this one has an open-source driver (however, it still has big closed-source RAID-on-a-card firmware burned onto the card, and a closed-source MegaCli tool for configuring the RAID). Dell PERC is cheap on eBay.

finally, if you can get rid of all your expee crap and use Macs, NFS works much better on Mac OS X than cifs, especially if you set it up like this:

http://web.ivy.net/~carton/rant/macos-automounter.html#9050149

it’s faster, more fault tolerant, case-sensitive, and more.

As your data was not on the boot drive and in a separate storage pool, this should be easy to regain access to your data. Assuming your storage pool was called ‘tank’ then type:

# zpool import -f tank

You will probably need the -f option as the pool was never exported before the system motherboard and boot drive died.

If you can’t remember the name of your storage pool, no worries, just type:

# zpool import

and Solaris will list all available pools.

When ZFS imports the pool, it should automatically remount all ZFS file systems that the pool contains. It’s possible you might get some warnings/errors if you changed mountpoints if they don’t exist, but you’ll easily be able to fix those problems if you see them.

Also, as you will probably need to re-create your Solaris user(s), you’ll need to see which user ids and group ids you used within your file systems and recreate the users and groups with the same ids. Something like this:

# groupadd -g 501 fred
# useradd -g fred -u 501 -s /bin/bash -d /export/home/fred -c fred-flintstone -m fred
# passwd fred

Explanation of useradd parameters used above:
-g fred: adds user to primary group ‘fred’ (which has groupid 501)
-u 501: creates the userid 501 for this user
-s /bin/bash: assigns the default shell to be bash for this user
-d /export/home/fred: defines the home directory
-c fred-flintstone: creates the comments/notes to describe this user as required
-m: creates the home directory for the user
fred: this is the login name for this user

Hope it helps.

Cheers,
Simon

Thank you,
Regards,
Adrian

My first question would be to see why you want to use iSCSI, when you probably just need a simple CIFS share, or Samba if you use a Linux client and it doesn’t have true CIFS client support.

If you think you really need iSCSI, then bear in mind that an iSCSI volume is a raw block device, so once the initiator connects to the target, you’ll need to format it using the file system you wish to use.

Hope this helps and is correct, as I had to cast my mind back a year to my experiences with iSCSI.

For my media centre/HTPC I am using a CIFS share of a standard ZFS file system, as described here:
http://breden.org.uk/2009/05/10/home-fileserver-zfs-file-systems/

Cheers,
Simon

I’m trying to build a ZFS target on Opensolaris essentially as a vault for my HTPC media collection. Also, although the (multiboot) HTPC primarily runs m$ Vista (spit!), I really want to get off m$ and onto Linux so my experimental iSCSI initiator is a Ubuntu 9.04 (Mint actually) boot.

Anyway, I think I’ve managed to create an iSCSI relationship between the two (after a battle to figure out the different syntax between Open-iscsi on Linux and on OpenSolaris), but the initiator seems to want me to fdisk the iscsi volume next, because it apparently doesn’t recognise what ZFS is, whereas what I was hoping to be able to do is to just mount the ZFS volume and rsynch the disk(s) on my HTPC to it.

Can someone please tell me:
1. The log /var/log/messages has some errors regarding IPv6 resolution and block size (10240) exceeding 1024 but I took these to be irrelevant as the iscsi target exists in the initiator when queried with ‘iscsiadm -m node’
2. Does the disk need to be specified as a raw device or something. I couldn’t figure out what circumstances would need me to specify a raw device rather than a disk.
3. Should I be able to do what I’m trying to do, or do I have to wait for something like Openfiler/FreeNAS to add support for ZFS before this is achieveable.
4. What about NexentaOS ?. I have steered away from this because as far as I can see, like WHS, it want to have control of all my disks.
5. Is my goal achieveable and what are my options ?
6. Who,what, where should I be asking this if not here.

Many thanks
Duncan

Thank you muchly for responding. To read “ZFS is all about data integrity” puts more of what I learned in context. It is all very impressive and probably a no-brainer for use in large data centres. I think I still have to think about it for home use though. This is a home backup server. Going to it to retrive files means something terrible has already happened. While something terrible could happen twice simutanouslty it is very unlikely and stuff so important that one must take such a castatrophe into consideration would be archived onto optical discs and stored away from the computer. However the fact one can spend the extra money on hard disks and get such a great level of protection is appealing just to say I did it.

Thank you again for responding to my post. I do have a question still though. If I have a “a collection of old, different-sized drives to form a large backup pool” that uses no redundancy like you do in one case and like I am contemplating and one of those drives dies do I loose all the data in the pool or just some of it?

(Also, I assume redundency could not be added to that pool latter as the disk sizes affect the set-up of RAID-Z and other redundency schemes. I suppose when I can obtain more disks I could set-up a new pool and copy the data over.)

Thanks a lot!

You can use ZFS with or without redundancy, the choice is yours. However, as ZFS is all about data integrity, it makes sense to use redundancy for helping to prevent data loss.

I have a fileserver that uses redundancy in the form of a single RAID-Z1 vdev, so that if one drive out of the four fails, the data will still be accessible, allowing me to replace the failed drive and rebuild it from the existing drives. In my case, the capacity of three drives is available for data, and the capacity of one drive is used for parity data (redundancy).

Redundancy is not wasting drives, although it may seem that way at first. The more redundancy (insurance) you build in to your data array, the more you are likely to avoid data loss. The amount of redundancy you choose to use depends on your budget or the importance of your data — i.e. how keen you are to prevent its loss.

I also have a backup server that uses no redundancy. It uses a collection of old, different-sized drives to form a large backup pool.

ZFS has many other advantages, including easy administration and providing seamless access to vast amounts of data without having to use clumsy volume managers. Also, snapshots are extremely powerful, allowing you to snapshot your data to perform full & incremental backups easily, roll back the file system to before your failed OS upgrade, recover previous edit states of any file, or even clone whole file systems.

ZFS also allows you to specify that a file system automatically and transparently creates 2 or 3 copies of every file within it. This may be useful for any data that is critical, such as user-generated content (photography, video, source code repositories, documents, spreadsheets etc). This means that ZFS can read the same file from multiple distinct locations off the drive surface in the event that, for example, bit rot has destroyed data within any file. This feature is called ditto blocks.

ZFS also allows hot spares to be included within your data array.

Using the above info, you have the ability to build a pretty much bullet-proof system. For example, you could build a storage pool consisting of a RAID-Z2 vdev, which uses the capacity of two drives for parity. You could also include a hot spare which ZFS will use automatically in the event of a drive failure. This means that your ZFS array can now rebuild itself automatically when a drive dies. Combining this array configuration with selected use of ditto blocks for critical file systems should create a system that should be extremely unlikely to ever lose any data. Then, to keep this system protected, you would setup automatic and frequent snapshots, send the differences between the snapshots regularly to a backup pool/server, and be sure to scrub the array with a frequency to suit the usage pattern of the system.

To see all of ZFS’ significant advantages, see here:
http://opensolaris.org/os/community/zfs/docs/zfs_last.pdf

I hope this answers your question.

Cheers,
Simon

While I realize you did not write the article as an invitation for support questions, it would be appreciated if you or someone could clarify some ZFS points for me.

Is ZFS only meant for high end use cases? I thought one advantage to using it was that you can add drives to a storage pool and it all becomes transparent to the user, just more storage space. So for example, four or five small hard drives could be connected together and it would act like one large drive. (One advantage of this would be that a very large file might be bigger than one of the smallest drives, but under ZFS that would be OK.)

Now I wonder if all my assumptions were wrong. It seems that one can use ZFS in a way that is great for data recovery but drastically reduces the storage space (RAID-Z, mirroring, etc), can use it to pool together many disks (as mentioned above) giving you great storage abilities, but the failure of one drive ruins all the data, or you can use it in a “normal” way (one disk=one pool) giving no
advantages at all.

Is there something I am missing about ZFS? It seems that there are no advantages unless one cares soley about redundency and has unlimited resources for disk drives. That is to say not really a home file server. (Not to say that just because the server is in the home does not make the data unimportant.)