Are you using NFS or CIFS to make the shares available? I seem to recall that NFS writes (synchronous IIRC) are slow but I’ll wait until you say which share method you are using.

Cheers,
Simon

I am slogging it out with my Solaris Express 11 box connected to a Windows 7 machine via Infiniband (drool). The storage pool on the Solaris machine has 3 X 2tb Deskstar 7K3000, I 5gb log and 30 gb cache (cache and log are both on a sandforce ssd). Reads are incredibly fast and over much more that 150MB/s but writes are depressingly slow at 30mb/s (max). I have tried disabling de-dupe (no luck) and setting log priority to through put but neither of these settings help much.

When I attempt a large write it starts of being quite speedy (100mb+) then drops off after a while (<10gb).

The system is running an i3 cpu with 8gb of ram. Running top show very little CPU usage (~20%), all but 800mb worth of ram but no swap.

Any advice would be greatly appreciated.

Kind Regards

Jim

On my Mac, I have setup autofs to mount my NAS CIFS shares using the following method:

Add the following line to /etc/auto_master :

/- auto_direct

Then create or edit /etc/auto_direct and include the following line for each share you wish to mount :

# photo library
/Users/simon/nas/photo/photo -fstype=smbfs ://simon:password@192.168.1.3/photo

In my home directory (/Users/simon) on the Mac, I created a directory called ‘nas’ and within that I created the ‘photo’ directory. This ‘photo’ directory is where autofs will create a mountpoint called ‘photo’.

I could have put all the mountpoints directly into the ‘nas’ directory, but this had the nasty effect that any access to the /Users/simon/nas directory via the Finder caused the Mac’s autofs to mount all CIFS shares listed in /etc/auto_direct, resulting in an annoyingly long delay before the Finder became responsive. The same problem occurred when opening or saving a file to the NAS from any application via the Finder open/save dialog window. Thus to eliminate this problem, I mounted each of the shares in separate individual directories, each one being one level below the ‘nas’ directory.

Later on, I had the idea to force autofs to mount all the shares when I logon to my Mac user account. I have a script which runs when I logon to my Mac user account. I called the script ‘mount_shares’ and it’s something like this:

#!/bin/bash

printf "\n" >> /Users/simon/log/mount.log
date >> /Users/simon/log/mount.log

printf "\nAttempt to mount photo:\n" >> /Users/simon/log/mount.log
ls -l /Users/simon/nas/photo/photo >> /Users/simon/log/mount.log

The ‘ls’ causes autofs to mount the share. Add a line like this to your ‘mount_shares’ script and then make the script run automatically when you login by adding the script to the ‘Login items’ of your user account in the ‘Accounts’ section of System Preferences.

Finally, in order to ensure the autofs mounts stay mounted longer than the default 10 minutes, edit the /etc/autofs.conf and modify the timeout to something like this:

Macintosh:etc simon$ cat autofs.conf
#
# This file is used to configure the automounter
#

# The number of seconds after which an automounted file system will
# be unmounted if it hasn't been referred to within that period of
# time.  The default is 10 minutes (600 seconds).
# This is equivalent to the -t option in automount(8).
AUTOMOUNT_TIMEOUT=360000

This causes autofs to mount the shares for 360000 seconds, or 100 hours.

Hope this helps.

Cheers,
Simon

# zfs set aclinherit=passthrough tank/home/fred/photo
# zfs set aclmode=passthrough tank/home/fred/photo
# chmod A=\
owner@:rwxpdDaARWcCos:fd-----:allow,\
group@:rwxpdDaARWcCos:fd-----:allow,\
everyone@:rwxpdDaARWcCos:fd-----:deny \
/tank/home/fred/photo
# chown fred:fred /tank/home/fred/photo
# zfs set sharesmb=name=photo tank/home/fred/photo

re this part:

“Within the Solaris file manager UI, you may need to set the attributes within the ‘Permissions’ and ‘Access List’ tabs of the properties for the /test/home/simon directory. After that, you may need to restart the Solaris machine (or probably just restart relevant services), and possibly the client machine to ensure it gets the new properties for the share.”

Is there any way to do that without the UI? My OSol box is not near a display

Did the smb config etc and nothing is showing up in my /shares directory …

I don’t regard myself as a UNIX/ZFS expert at all, but I’m a self-taught user of Solaris & ZFS, and have found that it does what I need very well, seems to be the best current storage solution, and it works — oh yes, and it’s free to those willing to learn and spend the time required to learn how to use it.

I can’t say whether your proposed hardware will definitely work or not, as I’m not familiar with the hardware.

You can check for your hardware on the Solaris HCL (hardware compatibility list).

However, 6 drives should be good for a data pool comprising capacity of 4 for data and 2 for parity (one RAID-Z2 vdev).

And 2 mirrored drives for a boot pool is also a good idea.

In general, Supermicro make excellent motherboards, and Intel chipsets are well-supported by Solaris.

As well as the HCL, try Googling for the various components including motherboard and solaris and see if there are any happy/sad users.

Just to answer directly your questions:
1. Yes, 2 pools are a good idea: (1) 2-drive mirror boot pool, and (2) multi-drive data pool using a single RAID-Z2 vdev for strong redundancy protection and simplicity of maintenance.
2. Six SATA ports for a RAID-Z2 vdev for your data pool are fine — see 1.
3. You need to verify Solaris has a driver for that graphics hardware — check the HCL or Google.
4. In general Intel ICHxx chipsets are well-supported by Solaris, as are their ethernet cards, although I have no personal experience with them, having gone for an AMD/NVidia setup with ECC memory.

Also, I would strongly recommend you use ECC memory, as ZFS is all about data integrity, and I don’t think you want to write garbage, due to corrupted flipped bits in memory, to disk.

If any of the ZFS stuff on this site helps you to get a good working ZFS system up and running soon, then feel free to thank me by clicking on a “Buy me a beer” button I shall add soon

Good luck!

Cheers,
Simon

I’m a user-level *nix user from a ways back, and I’m willing to do the brain work needed to get a server going, although I have no interest in using solaris or zfs as a hobby. I’m trying to build a tool, and willing to do the work to get it. I have worked in the computer industry for a couple of decades, just not in this particular sub-part. As an uninformed person in this particular niche, knowing only what I’ve read on line, could I get a quick glance over my shoulder for “Will this work??”

I’m decided on a system like this:
- 6 1-TB SATA disks for the main pool
- 2 mirrored disks to boot the OS
- Xeon E3110 processor
- Supermicro X7SBL-LN1 motheboard
- add memory to budget

What I’m looking for is some intelligent, experienced advice on the following questions, largely because I’m a wimp about tossing out $1K without first checking to see if I have some chance of success. These are all in the “I think it works, can you help me verify?” category.
1. Is the mirrored boot disks separated from the main pool workable?
2. Are six SATAs enough for raidz2?
3. Will the onboard XGI Z95 graphics serve to get the install/etc. done?
4. Does the Intel chipset (Intel 3200 + ICH9R + Intel 82573L)work with supported Solaris drivers?

I’ve been hacking on this for about a month, and I … think… this is a reasonable setup, but I’s sure like some expert advice about any hidden gotchas before I start counting out the cash for it.

I’d be eternally grateful, and owe any helpers much beer…

(in 10.6 you mount NFS via Disk Utility)

It seems OS 2008.11 and 2009.06 differ somehow in the CIFS sharing, but I now upgraded my OpenSolaris box to 2009.06 and I am happy again
The Snow Leopard Mini can once again read and write normally to the shares, everything looks normal again.

Cheers,
Jay

In that forum thread, some people are mentioning trying to connect to SMB shares on non-ZFS systems, and some also mention use of the Samba software application, just to make things more complicated. I say this because, from a ZFS perspective, ZFS does not use Samba, but CIFS instead, which is much better than Samba, as its implementation of the SMB protocol is exactly the same as Microsoft uses, so emulates a Windows box much more accurately than the Samba software. At least this is my understanding based on the CIFS reading I have done. It makes use of CIFS preferable to using Samba software. Unfortunately, it seems many people use the terms Samba, SMB and CIFS interchangeably and this can also cause confusion.

Another thing that becomes apparent is that in Snow Leopard, Apple have modified the way it calculates a file’s size. Now it seems they use the size on disk rather than the actual bytes consumed within the file itself, if the reports on that forum post are accurate… lots of separate problems seem be present, and I guess it will be a while before Apple is able/willing to devote resources to fixing all of these different issues. Glad I stuck with Leopard, and for anyone able to do so, it seems rolling back might be a sensible option. Unfortunately though, most people probably just upgraded their existing Leopard boot disk, so will be unable to “roll back”.

I’m still using Leopard, so I haven’t yet experienced this problem, luckily.

To me, it sounds like they changed the way the permissions are working through the Finder application.

If you have time, give this a try, as this could be an ACL issue. Create a test file system such as tank/test, share it using the usual methods and then create a file in it from the NAS, and then try to view the file from the Mac’s Finder application, then try to open the file via the Finder and edit the file, then try to save it. What happens? For setting a fully permissive ACL on the test file system, try the following, and let me know what you discover:

# zfs set aclinherit=passthrough tank/test
# zfs set aclmode=passthrough tank/test
# chmod A=\
owner@:rwxpdDaARWcCos:fd-----:allow,\
group@:rwxpdDaARWcCos:fd-----:allow,\
everyone@:rwxpdDaARWcCos:fd-----:allow \
/tank/test

cheers,
jay

So I have try to do it in Terminal, an error comes to me, said “ould not copy extended attributes to /Volumes/share/filename: Input/output error
The file actually wrote to the share, but seems the resource file cannot be create. I did some google search but no luck. You are one of the active blogger that using Mac + ZFS, so did you face this issue as well? Thanks for your feedback and advise.

Cheers,
Fai

So can you explain how you connect to your ZFS CIFS shares from your Mac, because this might be key to locating the issue.

Preferably, copy and paste the configuration lines that you use to access your shares, from /etc/auto* files.

Once you debug the CIFS access problem it would be interesting to hear from you. In the meantime, thanks a lot for the info — I’m sticking with Leopard for now

If Apple had moved to ZFS root boot, you could’ve just done a rollback to Leopard from Snow Leopard… with a one liner, well… assuming that Leopard itself had also used ZFS… come on Apple, time to move on up!

I didn’t have too much time to check it all out last night as I first tested the link to my ZFS server, which didn’t work, so I spent the night troubleshooting that!

Anyway. ACLs look good. Mine just grant everything to everyone!

To recap:

I cannot copy from Mac SL to the ZFS pool shared via smb. Get the -36 error.
I can delete files via Finder
I can create new folders via Finder
I can copy via Terminal
Perms etc look good on the file copied via Terminal from Mac and doing an ls -V on the Solaris box. I can read and write according to perms.
That file, when viewed in Finder cannot be opened, but can be played in the “Get Info” preview box
I can drag the greyed out file into iTunes
I can change metadata on that file via iTunes (i.e. write to the file)
I can play it through iTunes and AppleTV
I can write a file via TextEdit, for example, and it all looks good.
I can mount, read, copy, play from my MacBook on 10.5.8.
The time on my Solaris box had slipped to more than 5 mins from the time on my Mac, so I updated that. Can cause issues apparently.
I’ve turned the firewall off. No change.
Only error I can see in the Mac logs that might be related:

smb_maperr32: no direct map for 32 bit server error (0xc00000e5)

Not sure what that is. Google pointed me at the time issue, which I’ve fixed, but still get the error.

I found this:

http://spiralbound.net/2005/09/22/macintosh-finder-copy-to-samba-share-problem

Which seems to describe exactly what I am seeing.

Says that he set posix locking = no on his smb share and it was all good.

But I’m not sure what this means:

“Samba share is mounted over NFS on the server”

I’ve never really understood this output:

# sharemgr show -vp
default nfs=()
zfs nfs=()
zfs/sandbox/fs1 smb=()
sandbox_fs1=/sandbox/fs1

They have just set sharesmb … but why is nfs=() on? Is that what they mean by a samba share over NFS?

Anyway…

For that fix, you need to write an smb.conf file. i.e. not use sharesmb=yes via zfs commands. Not sure how they interact. i.e if you need to disable in zfs first?

I tried creating an smb.conf file and restarting the services. No idea how to tell if they read the new conf. So I rebooted. Now the mac wont connect at all. But, I’ve come to the conclusion that if you change anything on the server side, it’s best to reboot the mac.

By this time it was a little bit late!

To try:

zfs set sharesmb=off and test my smb.conf setup with public = yes first (read that this fixed one guys issue)
If that doesn’t work, try with locking off
Try using samba instead of smb. They are not the same apparently.
Finally, try NFS again, but couldn’t get that to work last night. It worked a year ago though.

-bash-3.2$ ls -V testfile1
-rwxrwx---+  1 fred    fred          0 May 10 21:30 testfile1
                 owner@:rwxpdDaARWcCos:------I:allow
                 group@:rwxpdDaARWcCos:------I:allow
              everyone@:rwxpdDaARWcCos:------I:deny
-bash-3.2$

or:

-bash-3.2$ ls -v testfile1
-rwxrwx---+  1 fred    fred          0 May 10 21:30 testfile1
     0:owner@:read_data/write_data/append_data/read_xattr/write_xattr/execute
         /delete_child/read_attributes/write_attributes/delete/read_acl
         /write_acl/write_owner/synchronize:inherited:allow
     1:group@:read_data/write_data/append_data/read_xattr/write_xattr/execute
         /delete_child/read_attributes/write_attributes/delete/read_acl
         /write_acl/write_owner/synchronize:inherited:allow
     2:everyone@:read_data/write_data/append_data/read_xattr/write_xattr
         /execute/delete_child/read_attributes/write_attributes/delete
         /read_acl/write_acl/write_owner/synchronize:inherited:deny
-bash-3.2$

More ACL info here:
http://breden.org.uk/2009/05/10/home-fileserver-zfs-file-systems/

Cheers,
Simon