Home Fileserver: ZFS File Systems

Once you’ve built your ZFS home fileserver / NAS, you’ll want to create your storage pool, create your file systems and share them to various devices around the home, such as laptops, PC’s, Macs, media centres etc.

I will go through all the necessary steps from start to finish, so you can see how to create a full, working file system hierarchy that is practical and useful.

Please notify me of any errors you may find in the comments section below.

Example scenario

Here I’ll describe an example scenario that could be reasonably typical.

Your NAS will serve family members and will consist of home directories and media file systems for music, photos and video, as in the following example:

home
    fred
        photo repository
        video repository
        projects
    wilma
media library
    music
    photos
    video

In this scenario, there are two users called Fred and Wilma.

Fred is a photographer and film maker, and so he manages photo and video repositories, and manages various miscellaneous projects too.

Note that Fred’s home directory file system will contain photo & video data which is irreplaceable, whereas the media file systems will contain data that can be replaced, either by re-copying from original CD’s or DVD’s, or re-exporting data from Fred’s home directory photo & video repositories.

I made this distinct separation to make it simple to understand and prevent mistakes: all media file systems are replaceable, and home directory file systems are not.

After a photo shoot, Fred will transfer all his photos from his digital SLR to the photo repository within his home directory. After editing the photos, he may choose to export selected photos to the photos file system within the media file system, using an export preset from his photo management software to make them available to other family members.

This separation enables different file system properties to be specified to reduce chances of loss for irreplaceable data, as we will see later.

I will also assume you wish to allow video and audio media clients around the house to be able to stream media from the media file systems within the storage pool, but this will be covered in a separate post for brevity here.

List your drives

ZFS allows you to create one big storage pool from all the drives you have in your system. The first step is to identify the drives:

# format
Searching for disks...done

AVAILABLE DISK SELECTIONS:
       0. c0d0 [DEFAULT cyl 20020 alt 2 hd 255 sec 63]
          /pci@0,0/pci-ide@4/ide@0/cmdk@0,0
       1. c1t0d0 [ATA-WDC WD7500AAKS-0-4G30-698.64GB]
          /pci@0,0/pci1043,8239@5/disk@0,0
       2. c1t1d0 [ATA-WDC WD7500AAKS-0-4G30-698.64GB]
          /pci@0,0/pci1043,8239@5/disk@1,0
       3. c2t0d0 [ATA-WDC WD7500AAKS-0-4G30-698.64GB]
          /pci@0,0/pci1043,8239@5,1/disk@0,0
       4. c2t1d0 [ATA-WDC WD7500AAKS-0-4G30-698.64GB]
          /pci@0,0/pci1043,8239@5,1/disk@1,0
       5. c4t0d0 [ATA-WDC WD7500AAKS-0-4G30-698.64GB]
          /pci@0,0/pci1043,8239@5,2/disk@0,0
       6. c4t1d0 [ATA-WDC WD7500AAKS-0-4G30-698.64GB]
          /pci@0,0/pci1043,8239@5,2/disk@1,0
Specify disk (enter its number): ^C
#

Here, ignoring the first drive, which is the IDE boot drive (drive number 0, id c0d0, marked as pci-ide), six SATA drives are shown (marked pci), whose ids range from c1t0d0 through c4t1d0.

I’ll use all of them to create a storage pool (zpool) in the following section.

Create the storage pool

Here you need to decide the storage pool vdev configuration. This determines the level of redundancy your storage pool will have.

Due to cost of drives, most people will choose RAID-Z1, and those who don’t want to take any chances will choose RAID-Z2. See the post Home Fileserver: A Year in ZFS for a more complete explanation — see the RAID-Z1 and RAID-Z2 headings there.

For a RAID-Z1 single parity setup, which can survive one drive failure like RAID 5, type:

# zpool create tank raidz1 c1t0d0 c1t1d0 c2t0d0 c2t1d0 c4t0d0 c4t1d0

For a RAID-Z2 double parity setup, which can survive two drive failures like RAID 6, type:

# zpool create tank raidz2 c1t0d0 c1t1d0 c2t0d0 c2t1d0 c4t0d0 c4t1d0

If you created a RAID-Z2 vdev for your storage pool, you should see something like this:

# zpool status tank
  pool: tank
 state: ONLINE
 scrub: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        tank        ONLINE       0     0     0
          raidz2    ONLINE       0     0     0
            c1t0d0  ONLINE       0     0     0
            c1t1d0  ONLINE       0     0     0
            c2t0d0  ONLINE       0     0     0
            c2t1d0  ONLINE       0     0     0
            c4t0d0  ONLINE       0     0     0
            c4t1d0  ONLINE       0     0     0

errors: No known data errors
#

Now let’s check the new storage pool:

# zpool list
NAME   SIZE   USED  AVAIL    CAP  HEALTH  ALTROOT
tank  4.06T   189K  4.06T     0%  ONLINE  -
#

So, we have 4TB of double parity storage available. But that’s before parity space is deducted.

So, in fact, we ‘only’ have 2.66TB of space for data, as the rest is for parity:

# zfs list
NAME   USED  AVAIL  REFER  MOUNTPOINT
tank   117K  2.66T  36.0K  /tank
#

Create the groups

We’ll now create two types of group: (1) groups for the users and (2) a group for media library access.

Create the media group for media pool access

# groupadd media

Create the primary groups for the users

Create the primary group that will be used by Fred. In this example, the fred group wants to use the id 501:

# groupadd -g 501 fred

Create the primary group that will be used by Wilma. In this example, we don’t care which id the OS gives us:

# groupadd wilma

Create the users

We’ll now create the user accounts for Fred and Wilma.

Create user for Fred

Fred wants to be able to log in to the Solaris NAS box with the bash shell.

Create the user ‘fred’, make his primary group ‘fred’, secondary group ‘media’, his shell ‘bash’, and set his password:

# useradd -g fred -u 501 -s /bin/bash -d /export/home/fred -c fred-flintstone -m fred
# usermod -G media fred
# passwd fred

Explanation of useradd parameters used above:
-g fred: adds user to primary group ‘fred’ (which has groupid 501)
-u 501: creates the userid 501 for this user
-s /bin/bash: assigns the default shell to be bash for this user
-d /export/home/fred: defines the home directory
-c fred-flintstone: creates the comments/notes to describe this user as required
-m: creates the home directory for the user
fred: this is the login name for this user

Create user for Wilma

Wilma does not want to log in to the Solaris NAS box, so she can have a simple account setup.

Create the user ‘wilma’, make her primary group ‘wilma’, secondary group ‘media’, and set her password:

# useradd -g wilma wilma
# usermod -G media wilma
# passwd wilma

Create the home directory file systems

Now we’ll create home directory file systems for two users: Fred and Wilma.

Fred is a Mac user, and Wilma is a Windows user.

Fred needs the photo & video repository and projects file systems to be accessible from his Mac.

Wilma needs her home directory file system to be accessible from her Windows PC.

Please note that I have used simplistic NFSv4 ACLs to make things simple here — i.e. all permissions allowed or denied at the owner, group and everyone levels. If you want to know more about this subject, take a look at the ACL chapter in the ZFS Admin Guide.

With the simplistic NFSv4 ACL property lists that I have used here, this will enable files and directories to be created or copied into these file systems, and these ACLs will be propagated via inheritance. See the brief ACL property list check section below, for user fred.

I don’t pretend to have expert understanding of this non-trivial subject yet, but these settings seem to have worked for me so far.

One non-desirable aspect of using this ultra-simplistic ACL is that non-executable files will still have the executable permission set. I probably should have specified ‘passthrough-x’ as the value for the ‘aclinherit’ property of these file systems. If someone more knowledgeable could comment on this point, that would be good.

Create the home directory file system

# zfs create tank/home

Create Fred’s home directory file systems

Create Fred’s home directory and child file systems for photo and video repositories, plus projects:

# zfs create tank/home/fred
# zfs create tank/home/fred/photo
# zfs create tank/home/fred/video
# zfs create tank/home/fred/projects

Note that I didn’t bother here making Fred’s home directory use the default Solaris location, which would be /export/home/fred, although this would be easy to do by issuing ‘zfs set mountpoint=/export/home/fred tank/home/fred’. Or you could have created the home directory at ‘/tank/home/fred’ and then set this when Fred’s user account was created.

Setup Fred’s photo repository file system properties

# zfs set aclinherit=passthrough tank/home/fred/photo
# zfs set aclmode=passthrough tank/home/fred/photo
# chmod A=\
owner@:rwxpdDaARWcCos:fd-----:allow,\
group@:rwxpdDaARWcCos:fd-----:allow,\
everyone@:rwxpdDaARWcCos:fd-----:deny \
/tank/home/fred/photo
# chown fred:fred /tank/home/fred/photo
# zfs set sharesmb=name=photo tank/home/fred/photo

This sets up inheriting of ACLs for files and directories using the ACL property list specified for owner (full permissions) and group (full permissions), and lastly for everyone (no permissions).
Also, the file system will be shared as a CIFS share using the name ‘photo’.

Setup Fred’s video repository file system properties

# zfs set aclinherit=passthrough tank/home/fred/video
# zfs set aclmode=passthrough tank/home/fred/video
# chmod A=\
owner@:rwxpdDaARWcCos:fd-----:allow,\
group@:rwxpdDaARWcCos:fd-----:allow,\
everyone@:rwxpdDaARWcCos:fd-----:deny \
/tank/home/fred/video
# chown fred:fred /tank/home/fred/video
# zfs set sharesmb=name=video tank/home/fred/video

This sets up inheriting of ACLs for files and directories using the ACL property list specified for owner (full permissions) and group (full permissions), and lastly for everyone (no permissions).
Also, the file system will be shared as a CIFS share using the name ‘video’.

Setup Fred’s projects file system properties

# zfs set aclinherit=passthrough tank/home/fred/projects
# zfs set aclmode=passthrough tank/home/fred/projects
# chmod A=\
owner@:rwxpdDaARWcCos:fd-----:allow,\
group@:rwxpdDaARWcCos:fd-----:allow,\
everyone@:rwxpdDaARWcCos:fd-----:deny \
/tank/home/fred/projects
# chown fred:fred /tank/home/fred/projects
# zfs set sharesmb=name=projects tank/home/fred/projects

This sets up inheriting of ACLs for files and directories using the ACL property list specified for owner (full permissions) and group (full permissions), and lastly for everyone (no permissions).
Also, the file system will be shared as a CIFS share using the name ‘projects’.

NFSv4 ACL property list check

As mentioned above, I have used very simplistic NFSv4 ACL property lists here for these file systems, and specified them so that these ACLs will be propagated into new files and directories that are created or copied into these file systems. To briefly show you what I mean, let’s create a couple of test files and directories to illustrate this:

# su - fred
-bash-3.2$ id
uid=501(fred) gid=501(fred)
-bash-3.2$ groups fred
fred media
-bash-3.2$
-bash-3.2$ cd /tank/home/fred/projects
-bash-3.2$
-bash-3.2$ touch testfile1
-bash-3.2$
-bash-3.2$ ls -l testfile1
-rwxrwx---+  1 fred    fred          0 May 10 21:30 testfile1
-bash-3.2$

Using ‘ls -l’ to view the standard Unix-style permissions, we see that ‘user’ and ‘group’ have read/write/execute permissions, and ‘other’ has no permissions. Note the ‘+’ (plus) character following the permissions list, which indicates that a non-trivial ACL is associated with the file.

Now if we use the ‘-V’ option with ls, we can see the ACL in compact format, where each letter represents a different permission:

-bash-3.2$ ls -V testfile1
-rwxrwx---+  1 fred    fred          0 May 10 21:30 testfile1
                 owner@:rwxpdDaARWcCos:------I:allow
                 group@:rwxpdDaARWcCos:------I:allow
              everyone@:rwxpdDaARWcCos:------I:deny
-bash-3.2$

These ACL entries were set when the file was created and have been propagated from the containing directory due to the ACL inheritance specified on the /tank/home/fred/projects directory above — see the ‘chmod’ statement above, and look for where ‘fd’ is specified, ‘f’ indicating file inheritance, and ‘d’ indicating directory inheritance.

Compact format ACL entries are perhaps easier to view than in verbose format, due to the fact that they align when viewing lists of these entries, like above.

However, it’s pretty hard to determine the meaning behind compact format ACL entries, so sometimes you may wish instead to view verbose format ACL entries, as in the following:

-bash-3.2$ ls -v testfile1
-rwxrwx---+  1 fred    fred          0 May 10 21:30 testfile1
     0:owner@:read_data/write_data/append_data/read_xattr/write_xattr/execute
         /delete_child/read_attributes/write_attributes/delete/read_acl
         /write_acl/write_owner/synchronize:inherited:allow
     1:group@:read_data/write_data/append_data/read_xattr/write_xattr/execute
         /delete_child/read_attributes/write_attributes/delete/read_acl
         /write_acl/write_owner/synchronize:inherited:allow
     2:everyone@:read_data/write_data/append_data/read_xattr/write_xattr
         /execute/delete_child/read_attributes/write_attributes/delete
         /read_acl/write_acl/write_owner/synchronize:inherited:deny
-bash-3.2$

Now we’ll make a test directory and create a file within it to observe the propagation of inherited ACL entries as nested files and directories are created within the file system:

-bash-3.2$ mkdir testdir1
-bash-3.2$
-bash-3.2$ ls -dV testdir1
drwxrwx---+  2 fred    fred          2 May 10 21:39 testdir1
                 owner@:rwxpdDaARWcCos:fdi---I:allow
                 owner@:rwxpdDaARWcCos:------I:allow
                 group@:rwxpdDaARWcCos:fdi---I:allow
                 group@:rwxpdDaARWcCos:------I:allow
              everyone@:rwxpdDaARWcCos:fdi---I:deny
              everyone@:rwxpdDaARWcCos:------I:deny
-bash-3.2$
-bash-3.2$ cd testdir1
-bash-3.2$ touch testfile2
-bash-3.2$ ls -V testfile2
-rwxrwx---+  1 fred    fred          0 May 10 21:40 testfile2
                 owner@:rwxpdDaARWcCos:------I:allow
                 group@:rwxpdDaARWcCos:------I:allow
              everyone@:rwxpdDaARWcCos:------I:deny
-bash-3.2$

NFSv4 ACLs are a fairly complex area, and I need to research the subject in more detail to get the most benefit from their power. Later… :)

Create Wilma’s home directory file system

# zfs create -o casesensitivity=mixed tank/home/wilma
# chmod A=everyone@:full_set:fd:allow tank/home/wilma
# zfs set sharesmb=name=home_wilma tank/home/wilma

This sets up inheriting of ACLs for files and directories using the ACL property list specified for everyone (full permissions).

Also, the file system will be shared as a CIFS share using the name ‘home_wilma’.

Permissions can now be set properly from an administrator account for Wilma on her Windows PC.
The wilma account must exist on the Windows PC. The PC is assumed to belong to a workgroup called WORKGROUP.

Domain accounts will probably require Active Directory administration, which is outside the scope of this post, but see the post on Active Directory for more details.

For reliable operation from Windows machines, check out the details on idmapping rules to map Windows user accounts to Solaris accounts. See here for more info: Solaris CIFS Administration Guide. See the Identity Mapping Administration chapter.

At some point, once I’ve delved into this subject in more detail, I will write another post. For now, the above should get you going for simple home setups.

Create the media file systems for music, photos and video

Now we’ll create the media file systems for music, photos and video.

Create the top-level media file system

# zfs create tank/media
# zfs set aclinherit=passthrough tank/media
# zfs set aclmode=passthrough tank/media

Create the music media file system and properties

# zfs create tank/media/music
# chmod A=\
owner@:rwxpdDaARWcCos:fd-----:allow,\
group@:rwxpdDaARWcCos:fd-----:allow,\
everyone@:rwxpdDaARWcCos:fd-----:deny \
/tank/media/music
# chown media:media /tank/media/music
# zfs set sharesmb=name=media_music tank/media/music

File system created in the usual way, owner and group set to media:media, and made a CIFS share named ‘media_music’.

Create the photo media file system and properties

# zfs create tank/media/photos
# chmod A=\
owner@:rwxpdDaARWcCos:fd-----:allow,\
group@:rwxpdDaARWcCos:fd-----:allow,\
everyone@:rwxpdDaARWcCos:fd-----:deny \
/tank/media/photos
# chown media:media /tank/media/photos
# zfs set sharesmb=name=media_photos tank/media/photos

File system created in the usual way, owner and group set to media:media, and made a CIFS share named ‘media_photos’.

Create the video media file system and properties

# zfs create tank/media/video
# chmod A=\
owner@:rwxpdDaARWcCos:fd-----:allow,\
group@:rwxpdDaARWcCos:fd-----:allow,\
everyone@:rwxpdDaARWcCos:fd-----:deny \
/tank/media/video
# chown media:media /tank/media/video
# zfs set sharesmb=name=media_video tank/media/video

File system created in the usual way, owner and group set to media:media, and made a CIFS share named ‘media_video’.

Setup CIFS sharing mechanism

First, as we will use CIFS shares to make the file systems available to Mac and Windows machines, do the following:

# smbadm join -w WORKGROUP
Successfully joined workgroup 'WORKGROUP'
#

Edit the /etc/pam.conf file to support creation of an encrypted version of the user’s password for CIFS.
Add the following line to the end of the file:

# vi /etc/pam.conf

other password required pam_smb_passwd.so.1 nowarn

Specify the password for existing local users.

The Solaris CIFS service cannot use the Solaris encrypted version of the local user’s password for authentication. Therefore, you must generate an encrypted version of the local user’s password for the Solaris CIFS service to use. When the SMB PAM module is installed, the passwd command generates such an encrypted version of the password.

# passwd fred
# passwd wilma
# passwd media

Restart SMB server and check shares

Now that your file systems are created and the shares have been specified, you’ll need to enable/restart the CIFS server on Solaris:

# svcadm enable -r smb/server
# svcadm restart network/smb/server:default
# svcs | grep smb
online         15:49:20 svc:/network/smb/server:default

Also, let’s check the shares are defined correctly:

# sharemgr show -vp
default nfs=()
zfs
    zfs/tank/home/fred/photo smb=()
          photo=/tank/home/fred/photo
    zfs/tank/home/fred/projects smb=()
          projects=/tank/home/fred/projects
    zfs/tank/home/fred/video smb=()
          video=/tank/home/fred/video
    zfs/tank/home/wilma smb=()
          home_wilma=/tank/home/wilma
    zfs/tank/media/music smb=()
          media_music=/tank/media/music
    zfs/tank/media/photos smb=()
          media_photos=/tank/media/photos
    zfs/tank/media/video smb=()
          media_video=/tank/media/video
#

Conclusion

When I get some more time, I’ll post details of how to access these shares from Macintosh and Windows systems. Hopefully, very soon… :)

In the meantime, you can take a look here for details written earlier on how to access these shares from a Macintosh computer. For accessing the shares from a Windows system, from the file manager, try the ‘Map network drive’ menuitem, giving the same share names as used above.

For more ZFS Home Fileserver articles see here: A Home Fileserver using ZFS. Alternatively, see related articles in the following categories: ZFS, Storage, Fileservers, NAS.

Popularity: 26% [?]

Share and Enjoy:

  • RSS
  • del.icio.us
  • StumbleUpon
  • Digg
  • Twitter
  • Mixx
  • Slashdot
  • Technorati
  • Facebook
  • NewsVine
  • Reddit
  • Google Bookmarks
  • LinkedIn
  • Yahoo! Buzz
  • email

37 Responses to “Home Fileserver: ZFS File Systems”

  1. Simon,

    you may run in problems, when denying ALL rights for group everyone (chmod A=everyone@:full_set:fd:deny). We encountered repeated access problems while working on shares from XP clients. This results in write/application errors on the client side even when you are owner and have owner@:full_set:fd:allow permissions on file and directory.

    Although I must admit that for the above reason I don’t exactly understand why, we keep at least the following allowed to avoid these problems
    everyone@:——a-R-c–s:——-:allow

    Furthermore on 2008.11 we encountered strange ACL inheritance behaviour when setting ACL rights on server side via chmod. We ended in
    1. doing full_set allows for @owner/@group/@everyone on CIFS share
    2. revoking (and afterwards setting) rights on client side

    I found hints to do so here: http://www.aspdeveloper.net/tiki-index.php?page=SolarisCIFSPermissions
    (Thank you “steveradich”!)

  2. Simon,

    I’ve looked around a little bit in ACL docs and (hopefully) understand the reason for my above mentioned problems:

    Windows ACL handling differs from ZFS: Windows first applies all DENY rules and only afterwards the ALLOW rules while ZFS handles ACLs in the given order. Thats the reason why you need to grant at least the above rights (aRcS) to everyone@ when using Windows clients.

    Otherwise your access may be denied before your intended ALLOW comes into place.

  3. Sorry, one more:

    At least you shouldn’t DENY (aRcs) for everyone@. It should work to leave these flags undefined in deny and allow for @everyone so later ACL entries can do the work…
    The last question would be how to finally deny these rights for unauthorized users?

  4. Hi Sebastian,

    Thanks for the info relating to ACLs for making CIFS shares reliably accessible from Windows environments.

    When I spend more time investigating access from Windows environments I will take your comments into account.

    When I was researching the ACL subject a couple of months ago, I found a very useful forum comment written by an experienced Windows systems administrator, and this is what he said (abbreviated):

    Ok, as a long term windows admin, I’m going to chip in with a couple of comments here.

    Firstly, I should make clear that I think the CIFS team have done a cracking job with this. An OpenSolaris file server works pretty much exactly like a windows fileserver, which is a huge improvement over Samba.

    We looked at Samba and found that it was completely unworkable in a Windows environment. OpenSolaris on the other hand allows us to keep our existing windows permissions and migrate them directly over.

    Windows and Unix treat deny entries very differently. Personally, every time I set up a CIFS share, I grant rights to everybody and from that point on do all my permission setting from windows. From the top of my head, the syntax is something like:

    # chmod A=everyone@:full_set:fd /path
    

    Yes, it gets complex if you’re working with files both in Windows and Unix, but that’s what the user mapping functionality is for.

    On the contrary, standard permissions working just like a windows server, and managable with the windows tools is *exactly* what I expected. Yes, there’s a bit of a learning curve to get OpenSolaris working, but I’ve been a windows admin for many years now, and I’m very, very impressed with this.

    Ross

    For more details, see the comment from myxiplx (Ross) @ Apr 9, 2009 5:27 AM here:
    http://opensolaris.org/jive/message.jspa?messageID=365620#365620

  5. Simon,

    Sorry if this is a dumb question but does your guide only work for Solaris 10 and not Open Solaris? I am trying some commands with Open Solaris 2009.06 and I’m encountering issues.

    The ls command doesn’t have the -V option

    $ touch test
    $ ls -V test
    ls: invalid option — V
    Try `ls –help’ for more information.

    The chmod command doesn’t seem to work.

    /tank/media$ chmod A=\
    > owner@:rwxpdDaARWcCos:fd—–:allow,\
    > group@:rwxpdDaARWcCos:fd—–:allow,\
    > everyone@:rwxpdDaARWcCos:fd—–:deny \
    > /tank/home/media
    chmod: invalid mode: `A=owner@:rwxpdDaARWcCos:fd—–:allow,group@:rwxpdDaARWcCos:fd—–:allow,everyone@:rwxpdDaARWcCos:fd—–:deny’
    Try `chmod –help’ for more information.

    Additionally when I copy files over from my OS X machine over SMB I get some weird permissions. The files show up as ———- permissions but I can still access them fine. This is a bit different from what I’m used to because if a file had that permission on Linux it would not be readable even by the owner.

  6. Hi there, my commands were run on an SXCE system, so it’s quite possible they changed the ‘ls’ and ‘chmod’ commands that appear on the default path on OpenSolaris.

    From memory, they changed the functionality of the ‘ls’ and ‘chmod’ commands to handle the ‘new’ NFSv4 ACLs to avoid having to use getacl and setacl commands, but the commands in the default path might be the ‘old’ non-NFSv4 ACL-capable commands, if you see what I mean.

    I have recently installed OpenSolaris 2009.06 so I’ll try out these commands a little later tonight when I have time. If you’re in a hurry though, check to see if multiple (probably two) versions of ‘ls’ and ‘chmod’ exist on your OpenSolaris system, and if so, run the commands found at the other location.

    Cheers,
    Simon

  7. OK, took a look as I was curious and my earlier comments seem to be true.

    If you login as a non-root user, and look where ls and chmod are, you’ll see that they are being found at /usr/gnu/bin:

    simon@blackhole:~$ which ls
    /usr/gnu/bin/ls
    simon@blackhole:~$ which chmod
    /usr/gnu/bin/chmod
    

    Whereas if you login as root, you’ll pick up these commands from /usr/bin:

    simon@blackhole:~$ su
    Password:
    simon@blackhole:~# which ls
    /usr/bin/ls
    simon@blackhole:~# which chmod
    /usr/bin/chmod
    

    Thanks for alerting me to this, I’ll have to fix my user’s path to use the more useful ones at /usr/bin which are capable of handling NFSv4 ACLs!

    Anyway, out of the box with the default OpenSolaris 2009.06 installation, as root you will have no problems running the commands listed above.

    Taking a look at the PATH environment variable for a non-root user we can see that /usr/gnu/bin precedes /usr/bin by default, so I suppose swapping them round should work, with hopefully no strange side-effects… :

    simon@blackhole:~$ env | grep ^PATH
    PATH=/usr/gnu/bin:/usr/bin:/usr/X11/bin:/usr/sbin:/sbin
    

    And for root, we see this:

    # env | grep ^PATH
    PATH=/usr/sbin:/usr/bin
    

    So root has no /usr/gnu/bin directory in its PATH.

  8. Thanks Simon, that worked perfectly. I was actually on root but I did ’su -’ instead of just ’su’.

    root@opensolaris:~# which ls
    /usr/gnu/bin/ls

    The ACL stuff is pure greek to me so I think I might stick to the regular permission setup for now. Your blog is a great resource for ZFS. Thanks for posting your experiences with it.

  9. Good news and thanks a lot. Yes, the ACL stuff is not trivial and I haven’t managed to find a very informative and simple explanation with practical non-trivial examples yet… soon I hope to dig deeper into that subject…

  10. Simon,

    Early in the post you seem to emphasize the relevance of the media filesystems as being replacable and the home filesystems as being irreplacable, but I don’t see any different in their creation except for access controls.

    Can you expand on how else they are treated differently, or at least tell me what I’m missing?

    Thanks

  11. Thanks Brad, you spotted my omission :) Yes, I forgot to complete that part. I think what I was going to do additionally to the irreplaceable user-created content file system was to set the ‘copies’ attribute to 2, so that ZFS would create two copies of each file stored in this file system to further reduce chance of data loss. Set the copies attribute to the value of 3 for maximum protection. Thanks for pointing that out!

    Cheers,
    Simon

    # zfs set copies=2 tank/home/fred/photo
    # zfs set copies=2 tank/home/fred/video
    # zfs set copies=2 tank/home/fred/projects
    
  12. Hi Simon,

    I set up something in a VM using your guide but I’m having trouble with samba. I use OS X 10.5 to access the shares I have set up which are tank/home/fyleow and tank/home/fyleow/music. I use finder to mount tank/home/fyleow and I can see the music directory, but I cannot read or write to it. In order to read and write to music I have to mount tank/home/fyleow/music separately. Any idea what might be the issue here? I don’t want to have to mount each file system under /tank/home/fyleow individually.

    I created the following file systems:

    zfs create tank/home
    zfs create tank/home/fyleow
    zfs create tank/home/fyleow/music

    Then I applied the following

    zfs set aclinherit=passthrough tank/home/fyleow
    zfs set aclmode=passthrough tank/home/fyleow
    chmod A=owner@:rwxpdDaARWcCos:fd—–:allow,group@:rwxpdDaARWcCos:fd—–:allow,everyone@:rwxpdDaARWcCos:fd—–:deny /tank/home/fyleow/
    chown fyleow:fyleow /tank/home/fyleow
    zfs set sharesmb=name=fyleow tank/home/fyleow

    zfs set aclinherit=passthrough tank/home/fyleow/music
    zfs set aclmode=passthrough tank/home/fyleow/music
    chmod A=owner@:rwxpdDaARWcCos:fd—–:allow,group@:rwxpdDaARWcCos:fd—–:allow,everyone@:rwxpdDaARWcCos:fd—–:deny /tank/home/fyleow/music
    chown fyleow:fyleow /tank/home/fyleow/music
    zfs set sharesmb=name=fyleow tank/home/fyleow/music

  13. Whoops a few typos in my post. That should read tank/home/fyleow/music and the chmod on the commands should be /tank/home/fyleow/music on the second set.

  14. Hi fyleow,

    I updated your original comment with your corrections.

    A couple of points:

    1. I presume you are using SMB/CIFS protocol from your client computer to connect to your shares, and not the actual Samba software which is a free software re-implementation of SMB/CIFS networking protocol?

    2. Also, it appears that the ‘zfs set sharesmb=name=fyleow’ within the two blocks above are identical. I presume this is a typo and you have, in fact, used different names for your shares — e.g.:
    zfs set sharesmb=name=fyleow tank/home/fyleow
    zfs set sharesmb=name=fyleow_music tank/home/fyleow/music

  15. Hi Simon,

    Sorry about that I did a hack job of copy and pasting. It turns out that what I’m describing is a known limitation in ZFS.

    If I create a series of file systems like your fred example and navigate to the parent fred file system over the share (/tank/home/fred) I can only have read/write access to that file system. I will still see the mounted child file systems (e.g. /tank/home/fred/projects) as folders but if I try to read/write to them I will get an access denied message.

    Anyway this bug report filed explains it much better than I can.

    http://bugs.opensolaris.org/view_bug.do?bug_id=6820940

  16. Hi fyleow,

    Thanks for the info. If you mount the shares for (1) tank/home/fred/projects, (2) tank/home/fred/photo & (3) tank/home/fred/video as three separate shares on the client OS then it works fine, at least it works fine for me.

    Simon

  17. Simon,

    fascinating ACL writeup, it looks daunting. In my shop they did it bass-ackwards and windows is the (normal) SMB server and NFS server. My sun boxes are the dummy nfs clients. I’m thinking of reversing the roles and making opensolaris the cifs/smb server, and nixing the nfs.

    But what I wanted to comment on, was “setting ‘copies’ attribute to 2″ ZFS. Dude – zfs snapshot trump copies=x. zfs clones trump snapshots. zfs SEND to another opensolaris server or offsite trumps clones.

    ps – your captachas are too hard!

  18. Hi Simon,

    Thanks for this great guide for a linux starter like me! I still have one problem: i created a user – like Fred. But everything Fred creates in the media-filesystem is not visible for other users (like wilma), because the files are owned by Fred and the group is also Fred (instead of the media group). How can i change this, so that everything Fred does in the media-filesystem is created with group media ?

    Many thanks,
    Bart

  19. Thanks Bart.

    When you boot your client OS, such as Windows, Mac OS or Linux etc, you need to connect to the shared media file systems using the ‘media’ user.

    Cheers,
    Simon

  20. All your ZFS stuff is very nice, thanks for this.

    I was just going over this post again, as I am currently changing my file system structure. What strikes me is that only in one place you create the file system with “casesensitivity=mixed”. Is there any particular reason for that?

    FYI: It seems that with snv_131 the “set sharesmb=name=xxxxx” is broken.

    Cheers,
    Christoph

  21. Thanks Christoph. It’s a while back now and I can’t remember, but I think I set “casesensitivity=mixed” when I was setting up the file system that was to be used for a Windows user.

    Cheers,
    Simon

  22. I have a estrange behavior on SVN_124 a go to OpenSolaris svn_134. I try set set a simple server for a hybrid MAC and windows XP on same LAN and access a a same server via CIFS and same configurations on your’s manual.

    But a have a issue on file is create from windows XP a initial set of attributes and ACL is set
    Ok

    id finance
    uid=104(finance) gid=101(jamute)
    groups finance
    jamute publico fdocs financ

    -rwxrwx—+ 1 finance jamute 49664 mar 23 2010 Ficha_SPOT_ALLIANZ_troca_loc2.xls
    owner@:rwxpdDaARWcCos:——I:allow
    group@:rwxpdDaARWcCos:——I:allow
    everyone@:rwxpdDaARWcCos:——I:deny

    Now a get access from Mac OS X 10.5 open a file on Excel:MAC 2008 modify a file and save in same name. the file continue ok attrib’s and acl

    -rwxrwx—+ 1 coordena jamute 49692 mar 23 2010 Ficha_SPOT_ALLIANZ_troca_loc2.xls
    owner@:rwxpdDaARWcCos:——I:allow
    group@:rwxpdDaARWcCos:——I:allow
    everyone@:rwxpdDaARWcCos:——I:deny

    try access again via Windows XP and modify a file again and save. but now a attributes are vanished and acl continue Ok.

    ———-+ 1 finance jamute 49975 mar 23 2010 Ficha_SPOT_ALLIANZ_troca_loc2.xls
    owner@:rwxpdDaARWcCos:——I:allow
    group@:rwxpdDaARWcCos:——I:allow
    everyone@:rwxpdDaARWcCos:——I:deny

    I can’t access a file from MAC.

    always access via CIFS.

    a zfs set is same on this manual

    ===================================================================
    raid5 aclmode passthrough local
    raid5 aclinherit passthrough local

    ===================================================================
    raid5/Financeiro aclmode passthrough inherited from raid5
    raid5/Financeiro aclinherit passthrough inherited from raid5
    raid5/Financeiro casesensitivity mixed -
    raid5/Financeiro sharesmb name=Financeiro local

  23. I resolve this issue configuring in this form :

    chmod -R A=\
    owner@:rwxpdDaARWcCos:fd—–:allow,\
    group@:rwxpdDaARWcCos:fd—–:allow,\
    everyone@:-wxpdDaARWcCos:fd—–:allow \
    /raid5

    I allow everyone for all users, if user is in valid group he can list a directory,

    I know this is not a best form to resolve this but a windows office 2007 and 2003 is is working fine now.

  24. Hi

    Does this sound like something you have come across before:

    When I mount my ZFS shares to my Mac over NFS – I can list and write. When I mount over CIFS – I can write but not list the current files.

    thanks

  25. Hi Shaky, no not seen that one before. I’m using CIFS shares and using ACLs as per above and haven’t seen this problem before.

    Cheers,
    Simon

  26. Very strange – I upgraded to 2009.06 (so much smoother than my previous experiences!)

    I’m going to create a new filesystem and try that out.

  27. It’s strange. This are my perms:

    james@dusky:/tank# ls -dV pool
    drwxrwxrwx+ 5 james james 8 May 2 21:12 pool
    owner@:rwxpdDaARWcCos:fd—–:allow
    group@:rwxpdDaARWcCos:fd—–:allow
    everyone@:rwxpdDaARWcCos:fd—–:allow

  28. It’s something to do with the perms and the old user. If I create a new filesystem, share, user (your fred example) it all works – and I get 45MB/s

    So something is up with my old pool permissions.

  29. I’ve not seen the problem you mention.
    For your old file system, your example shows what looks to be like all permissions allowed for owner, group and everyone.
    So perhaps you just had undesirable ACLs setup for this old file system? The fix might be to review what you want and then check it out by cloning the old file system, then setting the ‘right’ file system properties (aclmode, aclinherit etc), on the clone, then resetting the ACLs for each directory and file within the clone.

    Another slight possibility is that there’s a problem of zfs version for the old file system, but that sounds less likely than what I mentioned above.

    ZFS has pool versions and file system versions.
    See here for version information:
    http://docs.sun.com/app/docs/doc/819-5461/appendixa-1?a=view

    Let me know if you discover anything.

    Cheers,
    Simon

  30. I created a new filesystem, set the ACLs per your instructions, copied the files into the new fs and it all worked. Just took 30 hours to do the copy. Luckily I had enough spare space to duplicate.

    cheers

  31. Glad to hear it worked.

    Cheers,
    Simon

  32. Hi there,

    Firstly, thanks for the mountain of ZFS tips, has helped immensely.

    I’m in the process of building a NAS but firstly I’m testing this all out on a VM. I’m using Openindiana as I wanted native ZFS support.

    I’ve basically followed your steps above to the letter T however when I try and access the shares from either Mac OS X or Win7 by smb://media@storage/media or \\storage\media and typing in the username/password I can’t access the share.

    When I look at /var/adm/messages I get the following:
    Oct 29 19:31:37 storage smbsrv: [ID 138215 kern.notice] NOTICE: smbd[STORAGE\guest]: media access denied: guest disabled

    Why is it trying to access the share as guest even though I’m logging in with media?

    Cheers.

  33. Nevermind, it appears /etc/passwd and /var/smb/smbpasswd were out of sync. /var/smb/smbpasswd had the wrong uid as I’ve been creating/deleting users a bit through testing

  34. I’m curious why you chose a single raidz configuration over multiple zpools. You have your six drives available. Why not create a four drive raidz for the media library data, then a two drive mirror for the home directory data? I realize this imposes storage restrictions on how much space is available to each. Part of the reasoning behind this configuration would be that as a backup solution, adding a third drive to the mirror, and once resilvered, keeping it off-site.

  35. Hi there,

    i got a share like: “Create Wilma’s home directory file system”

    From my Windows box a can create, delete and rename files. I also can set and change permissions.

    But if I try to execute a programm on the share (example: fubar.exe), I always get something like: “Can´t access file… permission problem?”

    Is there perhaps something like noexec for the Windows box?

    :-(

  36. Hi Mike,

    Currently, I have one RAIDZ2 vdev within my storage pool, so 2 drives can die before losing any data – like RAID6 but better due to ZFS advantages already mentioned elsewhere.

    If I had done as you mentioned, I would have lost storage space if I had wanted to maintain the same level of parity – i.e. protection from loss.

    For example:
    1. If I created a 4 drive RAIDZ2 vdev for the media library, 2 out of the 4 drives would be used for data, and 2 for parity.
    2. 2 drive mirror for home directory, so 1 more drive lost to parity. And, importantly, this 2-way mirror has single parity, unlike the double-parity my existing setup has.

    So we have two pools, and three drives lost to parity, with your suggestion, with the additional disadvantage that the home directory can only suffer one drive loss before losing data.

    Whereas, with my existing setup:
    - only 2 drives lost to parity, so the capacity of one more drive is available for data.
    - the advantage of more flexible storage – either media can grow to fill all available space, or ‘home directory data’ can grow to fill all available space.
    - both media & home directory data share the same double-parity level of protection.

    I hope this is clear.

    Cheers,
    Simon

  37. Hi Sebastian,

    I did make some changes in permissions recently, so I will test out the scenario you mention and update the text if necessary. Thanks for drawing my attention to this.

    Cheers,
    Simon

Leave a Reply