Home Fileserver: ZFS setup

Note that on Leopard you can also use the Directory Utility (Applications > Utilities) to set the automounts graphically.

Do you have any performance issues if trying to watch .wmv, .mpg,… which are on the ZFS server on the Mac? I do have a lot…

Thanks for the compliments z0mbix !

@Alex Blewitt:

Thanks for the info. Yes, I think I also read somewhere about NFS waiting for the ‘data written OK’ ack before continuing. When I revisit NFS sharing I will take another look into this subject.

When using NFS sharing, I remember also getting very slow writes from the Mac to the ZFS fileserver on the Solaris box when I was using the Mac’s Finder app to drag and drop files to copy them to the fileserver. The speed was (sometimes?) abysmal (3-5MBytes/sec) when copying using the Finder. However, when I used the command line, I got a sustained 25-30MBytes/second, but I forget with which command: rsync or cp. So it seems there’s also some funny business going on with the Mac’s Finder app too.

The trouble with quoting speeds here is that I didn’t keep a copy of all my tests and setups, and my pool configuration was also changing, so to get definitive speed comparisons I would need to do this scientifically, but I don’t have the time right now 🙂

I was in a similar boat, setting up a fileserver, on existing hardware. This means I didn’t need to read the HCL! 🙂

I tried Nexenta (CD wouldn’t boot) and some Solaris flavours (OS wouldn’t boot) but then trialed FreeBSD, which worked beautifully.

I gotta say this ZFS business is one classy affair. After much meddling an continuous awe, I decided to go for Linux, RAID and LVM. The main reason for this was the adding of disks to increase a pool. I couldn’t believe this was not possible in ZFS when using RAIDZ devices!

With all the whizbang features of ZFS, you’d think this was a minor item to check for on the list.

It pained me to do leave ZFS behind, perhaps we’ll meet up for a fling in the future.

McP.

@McPop: Yes, I understand that Solaris is picky about the hardware due to relative lack of drivers. As you say, I think I heard that FreeBSD has wider hardware support, but I didn’t look much into it.

You are right — it’s a great pity that there’s no support yet for adding or removing disks from a ZFS RAIDZ vdev, although I expect they are working on this, but who knows when it will appear? The way I overcame this issue is to give my pool much more capacity than I currently need. Hopefully, by the time I need more space, they will have added the ‘add disk’ feature to RAIDZ vdevs. Alternatively, even without the ability to add disks to a RAIDZ vdev, you can ‘zfs send/receive’ your pool to another pool, like a large backup pool, and then reconfigure your pool by destroying it and recreating it with the new disk(s), then ‘zfs send/receive’ to your new larger pool. I didn’t try this yet though, but I think that’s possible to do.

Even if I did need more space now, I didn’t do the ‘zfs send/receive’ stuff, I could add new disks to the existing pool by adding a new vdev — i.e. a new RAIDZ or mirror vdev. Another alternative is to create a new pool with the new disks. So, you can see that there are already existing possibilities if you need more storage capacity, even before they add the new functionality to increase vdev capacity.

Oh, and there’s yet another possibility: replace each disk in an existing vdev with a larger capacity drive, one by one. So, for example, you could replace four 500GB drives, one by one, with four 1TB drives to double your pool’s capacity. You can then utilise the old disks for a backup pool or just another storage pool.

@Ruckus Ron: Thanks! That’s not an area I’ve really explored too much yet, but I believe it would be done using ZFS ACL. Go to opensolaris.org, click on Discussions and look into the storage or ZFS forums. Alternatively, click on Communities and look there under storage or ZFS. I’ll be looking there soon too 🙂 And you could look at the ZFS Admin Guide under the ACL section.

@Elvis: Using NFS I was getting quite disappointing speeds of around 25-30 MBytes/sec to and from the fileserver from the Mac Pro. When I switched to CIFS I got around 40+ MBytes/sec. Today I’m getting around 80 MBytes/sec with the same disks at each end, but by using dual Gigabit ethernet at each end and using an IEEE 802.3ad compliant Gigabit switch. See the page on Trunking for more details. However this will cost around $100+ for a switch that can do this. With this setup I’ve reached the limits of the disk read/write speeds on the Mac Pro and to go faster I’d need to get a faster disk like the Samsung Spinpoint F1, or to add an extra disk to the Mac and use a RAID 0 formation. This could potentially yield speeds of upto 150 MBytes/sec, but this is just a wild guess 🙂

CPU utilisation was minimal as I’m using a 64 bit dual-core AMD processor. Is the Sempron 64 bit or 32 bit? It seems the simple rule with ZFS is (1) give it loads of RAM (4GB is nice), and (2) use 64 bit processors.

@Elvis: I think the Athlon X2 will give you a lot more power than the Sempron — it’s what I use here and I never saw any issue with processor overload. At least 4GB of RAM should give a nice setup. If you are able to, try to use ECC RAM, as it can detect and correct errors before they’re even sent to the disk and it’s only about a 10% price premium. A no brainer as far as I see it. If you’re thinking of video editing then aggregating multiple gigabit ethernet (2+) links will certainly give a boost to your throughput. Glad you’re having fun getting it all working. I had a lot of fun getting it all working here too — sometimes more ‘fun’ than I bargained for 🙂 If you get time, let me know how it works out once you get your new gear installed and setup.

If you do link ethernet ports you’ll need an IEEE 802.3ad capable switch. I used the Linksys SRW2008 as it is reasonably priced and allows flexible aggregation possibilities, but its downside is that the web interface requires Internet Explorer, which I used in a virtualised Windows environment on the Mac. With this switch, a limited number of options can be set via telnet access. On hindsight, I should probably have bought an HP ProCurve 1800-8G switch, as it has very good feedback and I think it works with Firefox too, so you don’t need to use Windows.

Then you need to create a Link Aggregation Group (LAG) on the switch for the ethernet ports you wish to aggregate into a single fast link. E.g. On an 8-port switch, choose ports 1 and 2 for your fileserver and create LAG #1 for this and tell the switch that LAG #1 will use ports 1 and 2. Then select LACP (Link Aggregation Control Protocol) for the LAG. You’ll need to repeat this for the other machine where you’ll do your video editing, but this time create LAG #2 for this machine and use ports 3 and 4.

If it’s a Mac, then in the System Preferences Network Panel, click on the gear icon under the network interfaces and select ‘Manage Virtual Interfaces’. There you’ll select ‘Ethernet 1’ and ‘Ethernet 2’ ethernet ports to aggregate, or bond. Then you can use DHCP like normal etc. And if you’re using Parallels Desktop, you’ll probably need to disable the 2 network interfaces it creates as these may interfere with your aggregated link.

If you’re not using a Mac, then hopefully this comment will help someone else.

Have fun!

any luck with finding that hosting?

I don’t suppose you know how to setup Appletalk networking for the shares on the Solaris box? Using netatalk under linux it is fairly easy.

After Simon very kindly left a comment on my blog referring me to ZFS and his guide here, I’ve downloaded and tried SXCE on vmware. With no current knowledge of Solaris I’ve managed to setup ZFS and get it all shared nicely on my network. Thanks to Simon’s heads-up, I’m going to be migrating my Gentoo fileserver to Solaris on ZFS as soon as I get some new hardware delivered.

Thanks again Simon

I have a P4@2.4GHz and 1GB RAM. I get like 20MB/sec read/write speed. That is due to P4 beeing 32 bit. ZFS is 128 bit and doesnt like 32 bit CPU. With 64 bit dual core CPU you get speeds in excess 100MB/sec.

Hi all. I managed to get iSCSI working under Mac OS X 10.5.3 with the OpenSolaris 2008.05 Release. iSCSI with ZFS without authentication. As a client I used the globalSAN iSCSI Initiator 3.3.0.35 beta.

@ist: Good to hear. So now you can use your Solaris volume as a destination for your Time Machine backups of your Macintosh.

Well, Lol, you’re a bit late to the party, but sit down and take it easy 😉

As we wouldn’t want the marketing guys to get a bad reputation, how about you suggest they use GiB instead of GB in their HD marketing? Good luck! 😉

Hi Shaky, with ZFS no formatting is required, so you can create the pool immediately.

Hi Shaky, sorry to hear of your initial problems. I don’t have time right now to look at this in any depth as it’s late now, but to get you started, perhaps you might want to consider the following:

1. Regarding the slow speeds you’re getting across your network, I presume you are using a 100 Mbps ethernet link — i.e. you’re not using a gigabit switch / router? Also, if you ARE using gigabit ethernet, be sure you are using Category 5e or 6 ethernet cable on BOTH ends of your network connections — i.e. on Mac and ZFS server. To check your network speed settings, as root issue the ‘dladm show-dev’ command from the command line — see the output from my system below, which shows that I have dual gigabit links operational:

# dladm show-dev
LINK STATE SPEED DUPLEX
nge0 up 1000Mb full
nge1 up 1000Mb full

2. I was unhappy with the write speeds using NFS sharing, so I switched to using CIFS sharing, and this did improve write speeds quite considerably. Using gigabit ethernet, category 6 cables and CIFS sharing, I was getting around 40 MBytes/sec sustained transfer speeds with a 3-drive array using a RAIDZ1 vdev to form the storage pool. When I used link aggregation to link 2 ethernet ports at each end, the sustained speed rose to around 80 MBytes/sec.

3. Regarding AHCI, I can’t remember — will look tomorrow. Try turning it off. As I have an NVidia-based MCP chipset, my system is using the nv_sata driver, but I think you say you are using a different motherboard, so I can’t say any more right now.

Hope this helps for now.

Regards,
Simon

hmm, i see this in the messages log:

Aug 8 03:31:32 opensolaris ahci: [ID 405770 kern.info] NOTICE: ahci0: hba AHCI version = 1.20
Aug 8 03:31:32 opensolaris unix: [ID 954099 kern.info] NOTICE: IRQ21 is being shared by drivers with different interrupt levels.
Aug 8 03:31:32 opensolaris This may result in reduced system performance.

Hi Shaky, you might be able to plug the ethernet cable directly into the Mac’s ethernet port with a standard cable, or you might need to use a crossover cable. Then, unless your Mac is running a DHCP server service, you will probably need to manually set the IP address on the Solaris end. I would get a switch though if you want the higher speeds permanently.

If you plan to connect other machines to the server your best best will be to use a gigabit switch. I can recommend a DLink DGS-1008D which is about 70 euros. This is a simple, fast and cheap 8-port unmanaged gigabit ethernet switch — just plug in cables and go, nothing else to do. It also has intelligence built-in that detects which ports are in use, and only uses electricity according to what it needs to power. See more here:
DLink DGS-1008D green ethernet 8-port Gigabit switch

However, if you have 2 gigabit ethernet ports on each machine, you might want to aggregate the links to get increased speed. If so, you’ll need a managed switch, which are about 150 euros. See here for more info: Home Fileserver: Trunking.

Regarding AHCI, did you try turning it off in the BIOS to see if that works better? It’s not an area I know much about right now, but I think it’s disabled in my BIOS currently. Next time I reboot, I’ll take a look at my BIOS settings. There’s an AHCI wikipedia page, which says AHCI allows things like hot-plugging and NCQ.

BTW, did you check your motherboard for compatibility with OpenSolaris by looking at the hardware compatibility list (HCL) that Sun have on their website? It not, it might be worth checking.

Good luck!

OK, glad it’s working now. I never saw that issue with this motherboard and configuration. Have fun!

Ah, just the installer shows 32 bit. The actual installed solaris is 64bit.

Well, zeroing the disks helped a little. Can now get 5MB/s. Still not awesome!

I haven’t changed to IDE from AHCI yet.

I tried the CIFS share method you detail above, but cant seem to get the passwd command to work.

this bit:

# vi /etc/pam.conf

other password required pam_smb_passwd.so.1 nowarn

# passwd simon

I’ve edited the pam.conf file, but the passwd command executes the normal passwd binary i.e. usr/bin/passwd

“When the SMB PAM module is installed, the passwd command generates such an encrypted version of the password”

how do you install the PAM module? Do you need to restart something after editing the pam.conf file?

thanks

If you can ‘cd’ to the share directory, that sounds good. I presume you can also create a file and ‘cat’ it etc from the Mac?

If so, then it might just be a problem that the Mac’s Finder doesn’t seem to be aware of the share, so try to restart the Finder by holding down the ‘alt’ key and right-clicking on the Finder icon in the Dock. Then click the ‘Relaunch’ menuitem in the popup menu.

Hopefully that should do the trick.

An initial access to a share might take a few seconds as autofs does its business. Shares will also timeout after a while, potentially causing short delays when re-accessing a little-used share.

3 MBytes/sec across your network sounds slow, but if you’re going through a 100 Mbps router you won’t get over about 10 MBytes/sec. Are you going through your router or are you using an ethernet patch cable to link your two machines directly? And what category ethernet cable are you using?

I get this message for each pool disk, and you clearly see that it says sata, pci and NO ide:
Aug 10 17:02:57 solarisbox sata: [ID 663010 kern.info] /pci@0,0/pci1043,8239@5,1 :

Strangely, yours doesn’t mention sata, but does mention ide… doesn’t look right to me. Have you checked anything related to SATA in your BIOS?

As for the messages relating to shared interrupts, I have similar ones in /var/adm/messages so it’s probably OK.

So, if you want over 10 MBytes/sec, you’ll need to get a gigabit switch. Plug the switch into the 10/100 router. Plug both computers into the switch.

Secondly, check that both computers use at least category 5e ethernet cable (I use category 6, but 5e should suffice).

That will get you a bit further down the road to getting higher speeds across your network.

Then you might need to research how to get a SATA driver running nicely for your mobo’s SATA chipset from users with the same mobo, and as it’s on the OpenSolaris HCL, it should be possible, hopefully. Good luck!

Then I would use CIFS sharing (NFS write performance was not as good as CIFS).

Got it set up. Checked Gigabit at each end – all good. However didn’t use Cat 5e cable all the way (too short!). Going to try that tonight.

Still, it’s slow. About 2MB/s. Actually slower than it was yesterday – I changed the drive setup again back to AHCI from IDE.

Read tests using some dude’s script show about 60MB/s. Which isn’t too far off what other testers get (70-80MB/s). So it’s just write speed, to a ZFS pool shared over NFS. Actually the NFS doesn’t really matter, the write speed is just slow.

As mentioned above, this guy has same hardware and same problem, but his fix didn’t work for me:

https://www.opensolaris.org/jive/thread.jspa?threadID=57647

Options:

0. Try the gigabit cable, but I dont think this is the problem now.
1. Try Linux, see if it’s the drivers. Could try sofware RAID or my mobo supports hw RAID. Don’t know if Linux would drive it though.
2. Go back to shop, change mobo and maybe processor if needed. Friend has Asus board that worked just fine. Super fast write speeds.
3. Keep fiddling. There must be a way to get the drives to work faster…. it’s a supported mobo and SATA interface.

I have to get it sorted. I can’t concentrate at work .. I’m just spending time reading forums!

But then I re-read your previous posts in this series:

“He says: “I’m using it [ZFS] because I’m fed up with losing data to weird RAID issues with Linux, and I believe that OpenSolaris with ZFS will be substantially more reliable long-term.”

Ah, I really do want ZFS

Yup, cat 5e all the way now.

Not really any useful info on Google for my mobo and solaris.

Going to try to find an SATA card and try that.

Oh, I just read that on some chipsets, when there are more than 4 SATA ports on the board, only the first four are usable properly. The others are hidden behind some ‘raidgoofyness’.

Mine are plugged in to 1,2,3 … and for some reason 5.

Could try 4!

Screw it. I spent. Pick up new mobo and cpu tomorrow.

Well, sorted, almost.

I got the Asus M2N-VM HDMI. Rebuilt the machine and installed OpenSolaris svn_93. No network. Apparently the nVidia chipset doesn’t work via the Solaris drivers. I tried the nfo 3rd party drivers but couldn’t get those to work. So I popped out and bought an Intel PCI gigabit ethernet card. All good.

BTW – my friend gave me the wrong mobo model name! He had the M2A-VM

Created and shared zfs pool via NFS and CIFS. I get about 20MB/s over NFS and 30MB/s over CIFS. I’m not using my new 5e cable yet though (need the long cable so I can have box in living room plugged into TV)

So I decided to go via CIFS. I couldn’t get the /shares mount to show up in Finder like you, so I mounted in /Network/Servers via the fstab file. Lovely, worked fine.

My user/group was set properly, however, any files copied via Finder to the share were written without any permissions at all on Solaris box.

I noted my zfs pool had aclinherit=restricted whereas yours has ‘secure’ I tried changing that, restarting the smb service, but the share just disappeared. Mac log said it was dead.

So .. still trying!

Tried svn_95. Now getting 35-40MB/s. Didn’t read all the comments, someone mentioned a problem with svn_93 and CIFS.

Fast enough for me.

Finally.

Many thanks for all your help during my ordeal!

Where can I find good info on choosing a CPU for OpenSolaris?

I want a dual core, low wattage, 64 bit CPU.

Should I go Intel or AMD?

Forget price/watt/performance ratios.

Which CPU would perform best in Opensolaris?

Hi,

I am getting ready to put together a computer (well, all of the parts that i’ve purchased over a few months) that will be my ZFS file server/media server. What are the main differences between SXCE and opensolaris 2008.05? Is one better/easier to use than the other?

Are there any good books out yet pertaining to ZFS or opensolaris?

Thanks,

Brian

hey Simon,

Thanks for your help. I have another technical question, as the computer is built(hardware wise). I have an
-asus M3N78 – PRO mother board
-AMD 2.6ghz X2 5000+ CPU
-4 GB G-Skill ddr2 RAM
-4 x 1500 GB seagate drives
-1 X 250GB seagate drive (BOOT disk)
-650W PSU
-case
-DVD-RW

My question is this: The MB has 6 SATA ports. I have three modes it can be in: AHCI, RAID, or IDE. The only way that the # 5 and 6 ports can be seen are if it is in AHCI or RAID mode (i need them for all the discs). I do not want to RAID the disks, as I will be using ZFS. and I am not sure what AHCI is. But SXCE (b99) (or opensolaris) will not boot in either mode, It hangs after I choose the SCXE install option and it says ‘SunOS release 5.11 Version….’. Do you have any ideas?
I can get opensolaris to install in AHCI mode, but then, randomly, I will get an error saying “ahci_port_reset: port (x) BSY/DRQ still set after device reset port_task_file = 0x180”.
port (x) is whatever port is geting the ahci error. The drives are fine, i ran seatools long test on them all. Is this a driver issue?

thanks.

Hi Brian, glad you managed to sort the problem out. I’m not sure about the ‘solaris express xVM’ option you mention as I’ve not used it. However, it does sound strange that the standard option makes the machine reboot! I assume you checked the Sun hardware compatibility list (HCL) before choosing the hardware, or found other people reporting success with Solaris and ZFS?

Regarding the sharing, I used CIFS and to get the sharing working, check the following sections above:
– Making the storage pool accessible from other machines
– Update 30/03/2008: extra steps required for CIFS setup
– Configuring client machine access to the file system share

If it still fails then see if you can ‘tail -f /var/adm/messages’ and then try to connect to the share from the Mac again and see if any failure messages are displayed on the console. That’s how I found out why my earlier attempts to get it working were failing. I saw this on the console:

When I try to access the share from the Mac using autofs (smbfs:), I see the following message in /var/adm/messages:

Mar 27 20:38:48 solarisbox smbd[667]: [ID 653746 daemon.notice]
 SmbLogon[WORKGROUP\simon]: NO_SUCH_USER

Also, from memory, I use AHCI, and this should enable hot swap of a failed drive, in theory.

Good luck, and let me know if you (1) get the connect working and (2) if you solve the boot issue.

Cheers,
Simon

Simon,

Thanks for the great help. I was able to get NFS working, i might work with CIFS later on but for now ill stick to NFS. I also have been able to get a shared folder up for everyone to read/write to which is nice. I’m slowly transferring (over rsync) close to 700 GB of data from my iMac and external FW drive to the zpool to give the drives and the machine a workout. So far it has been on for about a day now and the fans on the case are still pushing out cold air! the hard drives are cool to the touch and i’ve had it transferring this data all morning and into the afternoon. I even tried it as a scratch disk for Final Cut Express and it worked flawlessly editing HD footage from my canon HF10. What a machine!

I have not had any time to diagnose the boot problem, but i am able to do everything booting to xVM so i will use it for now. Next step after i get all the data over and attempt time machine backups, i’ll work on getting an ubuntu virtual machine up for some media streaming. Do you know of any good media streaming programs I could use under SXCE to stream to a PS3 and xbox360? This is one of the goals of this machine, but im sure it will be further down the road.

Thanks for the help again and the great articles. Between here and the solaris forums I’ve come from having never used solaris before to feeling quite comfortable in it in about a weeks time. So far I am loving it.

brian

I wonder, would you mind writing a new article for your excellent ZFS series? (BTW, I have posted links to your site on several Linux/Solaris forums). In the article I would like you to wrap up your experiences, now that you have used ZFS and solaris for a while, as someone coming from Linux. I think many would like to read a thorough article on that, just as all your other well researched articles. Are you a convert? If yes, why? If no, why not? Etc. For the coming flame wars between Solaris and Linux, I can always link to your site as someone who has tried both. ;o)

I dont mind if you delete this post. I just wanted to ask you this question. You can delete this post, and just write such an article. Pliiiz?

Hi Kebabbert,

thanks again and I think that would make an interesting post, like you say.

I had been thinking of writing a post like this for a while, but due to my pedestrian immersion into the world of Solaris and ZFS etc, I have still not really grappled successfully with all of the areas of interest relating to this fileserver/NAS. So I still have a few areas to explore or return to, and I will write some posts on these subjects soon. After that, I will write the kind of summarizing post that you mention.

Thanks for posting links to these ZFS articles — I had noticed sudden huge increases in web traffic, mostly from Sweden and other Scandinavian countries 🙂

One thing though — I’m not really a die-hard Linux user, but I ran it at home for a few years. I did like using Linux a lot after I stopped using Windows at home around 2000. I used a Linux LAMP configuration to run this website prior to moving it over to a Solaris SAMP configuration. Around 2004 I bought a PowerMac and then Mac OS X became my preferred OS for home use. I found its apps and especially the multimedia apps had great usability and made using them a pleasure. Of course, with the Mac’s command line I could continue with using UNIX commands. If of interest, I wrote a brief comparison of Windows, Linux and Mac OS X here (bear in mind that it was written a couple of years ago and Linux is always improving, so…):
http://breden.org.uk/2006/11/12/windows-linux-and-mac-os-x-shootout/

Personally, as I am involved in photography (macro, event, family etc) I find the Aperture 2.0 software running on Mac OS X is a superb piece of software that helps me organise my big photo library, which is growing rapidly due to shooting in RAW format.

Then there’s GarageBand (low end), Logic Express (midrange) and Logic Studio (high end) for music production.

And for managing video productions you have great software from iMovie/iDVD (low end), Final Cut Express (midrange) and Final Cut Studio (high end).

And for software development, a Mac is great too.

So for me, for most of my needs, Mac OS X is my OS of choice on the client side, and Solaris for the server side: 1. ZFS fileserver/NAS, and 2. web server.

Cheers,
Simon

Simon,
I’ve upgraded to the latest version of opensolaris 2008.11 and i’d like to keep my old ZFS pool. I install the OS on a separate disc than the others that the zpool is on, so I know the pool is still there. How do I mount the old zpool, and then get it to mount every time I boot up the machine?
thanks.

Awesome! It worked perfectly.
have you tried the new 2008.11? I’m enjoying the slider to control snapshots. I like using the command line, but the gui that time slider adds to snapshots is neat and easy for anyone else in my home to use if they need to.

Thanks!

Simon,
I’m getting an error on my zpool. It does not seem fatal, but I cannot pin down exactly what it means. Have you had this before? I set it to scrub. I am currently creating an image of my imac (450GB) to the server, because I need to do an archive and install on the imac. Could it just be that there was an interruption in the data flow? Is that all the error means and I just need to do a zpool clear?
Thanks.

pool: chuck
state: DEGRADED
status: One or more devices are faulted in response to persistent errors.
Sufficient replicas exist for the pool to continue functioning in a
degraded state.
action: Replace the faulted device, or use ‘zpool clear’ to mark the device
repaired.
see: http://www.sun.com/msg/ZFS-8000-K4
scrub: scrub in progress for 0h2m, 0.43% done, 8h41m to go
config:

NAME STATE READ WRITE CKSUM
chuck DEGRADED 0 0 0
raidz1 DEGRADED 0 0 0
c3t1d0 ONLINE 0 0 0
c3t2d0 ONLINE 0 0 0
c3t3d0 ONLINE 0 0 0
c3t4d0 FAULTED 5 12.0M 0 too many errors

errors: No known data errors

Simon,
Unplugging and replugging the SATA and power cables did the trick to rid of the read/write errors. Now I’m having a checksum error on the same disk. I had time slider turned on, but i’d like to delete the snapshot taken of a 450GB .DMG file because i believe this file is corrupted (per the read/write error earlier). I’m now getting this:

pool: chuck
state: ONLINE
status: One or more devices has experienced an unrecoverable error. An
attempt was made to correct the error. Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
using ‘zpool clear’ or replace the device with ‘zpool replace’.
see: http://www.sun.com/msg/ZFS-8000-9P
scrub: none requested
config:

NAME STATE READ WRITE CKSUM
chuck ONLINE 0 0 0
raidz1 ONLINE 0 0 0
c3t1d0 ONLINE 0 0 0
c3t2d0 ONLINE 0 0 0
c3t3d0 ONLINE 0 0 0
c3t4d0 ONLINE 0 0 31

errors: No known data errors

Can you tell me how to locate and destroy the snapshot taken? (I have deleted the actual file already, but the snapshot remains, taking up 450GB of space)

Thanks for the help.

Simon,

I’ve built this monster now. I’ve got the RAIDZ pool up and running and I can access the data remotely. This would seem like a success but I have many problems that I was wondering if you could assist with some?

1. I can’t access the Java Web Administration console from any machine except localhost (even though my firewall is disabled.

2. Using the Samba shares the user priviliges are obeyed but group priviges are not. Is this a samba limitation? I tried NFS too but couldn’t figure out how to set custom share names. Why do you use Samba as apposed to NFS?

3. Any data that I create locally on the server in these file systems has incompatible permissions as cannot be accessed.

the other problems are issues with things that you have not attempted to cover here so I won’t bring them up.

Thanks,
Kev

Recently i found that there is an error 50 appear when using QuickTime to watch video file from a Windows XP to access the ZFS CIFS share.
some people suggest that adding
mangled names = no
in smb.conf
however, it required to use Samba instead of Solaris CIFS.
Did anyone try any alternative?
Thanks for any advise

I am using Solaris Express build 105
Actually the CIFS works well except opening the video file using QuickTime from Windows (using Mac is alright)
All Quicktime is latest version.
And i am able to watch video file using Windows Media player and listen to mp3 files as well.

I’ve did a research and no idea too.
Anyway, thanks for your help
🙂

Hi Simon,
Thanks for your suggestion. However, the error occur only when the file is located on a Solaris CIFS share, its working when the file is located on a Windows Share.
Really thanks for your advise.

Cheers,
Fai

I’ve installed OpenSolaris 2008.11 onto a 160GB using the full capacity of the drive. I have three 250GB drives. When I run the ‘format’ command, those three 250GB drives say “unknown drive type”. They were pulled from working systems (two had windows on it). I deleted the partitions and formatted them to Solaris2 (and they are not active partitions). Now, (and even before I did this), I am getting a message after trying to do #zpool create tank RAIDZ1 .. .. .. that says this: cannot open ‘RAIDZ1’: no such device in /dev/dsk must be a full path or shorthand device name. Oh, I should note that I logged in as the user I created, but then did a “su -” and entered the root password. I am totally new to Solaris/unix so maybe I did something wrong. Any help is appreciated greatly!

Oops, thanks James & Kristian, I fixed that now — it should have been lower case 🙂

Hi Kristian,

Sounds like the group has not been created. Try logging in as kristian and then typing ‘id’ — see below:

-bash-3.2$ id
uid=501(kristian) gid=501(kristian)

You should see something like the stuff above. The values are not important, but the ‘gid’ entry must be present.
If you have not yet created the group called kristian, then to create the group and make the group the primary group of user kristian, type:

# groupadd kristian
# usermod -g kristian kristian

To change the user & group for a file at the same time, try:

-bash-3.2$ chown kristian:kristian filename

or to change the user & group of all files and directories recursively, try this:

-bash-3.2$ chown -R kristian:kristian *

Hope it helps.

Hi Kristian, try running the groupadd command as the root user (su).

Hi … I’m back 🙂

Have you upgraded to Snow Leopard? Any issues?

I’ve upgraded and can no longer write to my SMB share. I get an error -36.

Can delete files. Cannot write.

Can copy via Terminal, but then cannot open the file I copied via Finder (it’s greyed out). If I add that file to iTunes, I can edit metadata and can also play in iTunes. Just seems like Finder isn’t working right.

Still investigating….

First impressions are good. I like the ‘intra-app’ expose. Seems fast. Mail is super fast.

I didn’t have too much time to check it all out last night as I first tested the link to my ZFS server, which didn’t work, so I spent the night troubleshooting that!

Anyway. ACLs look good. Mine just grant everything to everyone!

To recap:

I cannot copy from Mac SL to the ZFS pool shared via smb. Get the -36 error.
I can delete files via Finder
I can create new folders via Finder
I can copy via Terminal
Perms etc look good on the file copied via Terminal from Mac and doing an ls -V on the Solaris box. I can read and write according to perms.
That file, when viewed in Finder cannot be opened, but can be played in the “Get Info” preview box
I can drag the greyed out file into iTunes
I can change metadata on that file via iTunes (i.e. write to the file)
I can play it through iTunes and AppleTV
I can write a file via TextEdit, for example, and it all looks good.
I can mount, read, copy, play from my MacBook on 10.5.8.
The time on my Solaris box had slipped to more than 5 mins from the time on my Mac, so I updated that. Can cause issues apparently.
I’ve turned the firewall off. No change.
Only error I can see in the Mac logs that might be related:

smb_maperr32: no direct map for 32 bit server error (0xc00000e5)

Not sure what that is. Google pointed me at the time issue, which I’ve fixed, but still get the error.

I found this:

http://spiralbound.net/2005/09/22/macintosh-finder-copy-to-samba-share-problem

Which seems to describe exactly what I am seeing.

Says that he set posix locking = no on his smb share and it was all good.

But I’m not sure what this means:

“Samba share is mounted over NFS on the server”

I’ve never really understood this output:

# sharemgr show -vp
default nfs=()
zfs nfs=()
zfs/sandbox/fs1 smb=()
sandbox_fs1=/sandbox/fs1

They have just set sharesmb … but why is nfs=() on? Is that what they mean by a samba share over NFS?

Anyway…

For that fix, you need to write an smb.conf file. i.e. not use sharesmb=yes via zfs commands. Not sure how they interact. i.e if you need to disable in zfs first?

I tried creating an smb.conf file and restarting the services. No idea how to tell if they read the new conf. So I rebooted. Now the mac wont connect at all. But, I’ve come to the conclusion that if you change anything on the server side, it’s best to reboot the mac.

By this time it was a little bit late!

To try:

zfs set sharesmb=off and test my smb.conf setup with public = yes first (read that this fixed one guys issue)
If that doesn’t work, try with locking off
Try using samba instead of smb. They are not the same apparently.
Finally, try NFS again, but couldn’t get that to work last night. It worked a year ago though.

Looks like Snow Leopard has changed some security aspects, esp. in Finder, because I assume you changed nothing on your Solaris ZFS NAS, just migrated from Leopard to Snow Leopard?

Once you debug the CIFS access problem it would be interesting to hear from you. In the meantime, thanks a lot for the info — I’m sticking with Leopard for now 😉

If Apple had moved to ZFS root boot, you could’ve just done a rollback to Leopard from Snow Leopard… with a one liner, well… assuming that Leopard itself had also used ZFS… come on Apple, time to move on up!

I didn’t ever set that up. Just used Finder->Connect to server.

Hi Simon,
Greeting from Hong Kong again. Recently, I’ve upgrade my MacBookPro to 10.6 (more specific 10.6.1). But I found that I am not able to write files to my ZFS file server via SMB anymore. I can mount it, read it, but when i try to write files to the share via Finder, it said the disk cannot be write. But actually I can create folder, and save file to there via other program (e.g. TextEdit)

So I have try to do it in Terminal, an error comes to me, said “ould not copy extended attributes to /Volumes/share/filename: Input/output error
The file actually wrote to the share, but seems the resource file cannot be create. I did some google search but no luck. You are one of the active blogger that using Mac + ZFS, so did you face this issue as well? Thanks for your feedback and advise.

Cheers,
Fai

hey simon, guys.
long-time listener, first-time caller 😉
i’m having the same problems with opensolaris shares (OS 2008.11), 10.5.8 works, snow leopard (10.6.1) fails.
Mounting a share from the OS server works ok, i can browse and read files but when i try to copy a file to the share i get
the “error -36” from the finder as well as “smb_maperr32: no direct map for 32 bit server error (0xc00000e5)” in the console. and it leaves a greyed-out “foobar.txt” file with 512 bytes size.
copying via terminal on the other hand works, no problems there.
things like setting the “DSDontWriteNetworkStores” default or mounting with “cifs://” instead of “smb://” make no difference.
i also tried switching to NFS, but no joy either – same symptoms as with cifs sharing – the mounted share can be browsed and read (even opening and editing an existing file works), but copying a file to the share via finder fails with the “-36”.
again, copying via terminal works.
shaky, you said NFS works for you? could you share your settings?

cheers,
jay

Well guys, it seems there’s a long thread on this problem of accessing SMB/CIFS shared files via Snow Leopard’s Finder application here: Topic : SMB -36 ioErr when opening files.

In that forum thread, some people are mentioning trying to connect to SMB shares on non-ZFS systems, and some also mention use of the Samba software application, just to make things more complicated. I say this because, from a ZFS perspective, ZFS does not use Samba, but CIFS instead, which is much better than Samba, as its implementation of the SMB protocol is exactly the same as Microsoft uses, so emulates a Windows box much more accurately than the Samba software. At least this is my understanding based on the CIFS reading I have done. It makes use of CIFS preferable to using Samba software. Unfortunately, it seems many people use the terms Samba, SMB and CIFS interchangeably and this can also cause confusion.

Another thing that becomes apparent is that in Snow Leopard, Apple have modified the way it calculates a file’s size. Now it seems they use the size on disk rather than the actual bytes consumed within the file itself, if the reports on that forum post are accurate… lots of separate problems seem be present, and I guess it will be a while before Apple is able/willing to devote resources to fixing all of these different issues. Glad I stuck with Leopard, and for anyone able to do so, it seems rolling back might be a sensible option. Unfortunately though, most people probably just upgraded their existing Leopard boot disk, so will be unable to “roll back”.

Thanks for the heads up Jay! So for any others who had this problem — if you’re using Snow Leopard, also upgrade your NAS to use OpenSolaris 2009.06 and everything should work fine. Good to know I can upgrade to Snow Leopard one day and not see this problem.

Jay – for NFS, I didn’t do anything other than use Disk Utility to mount the server

(in 10.6 you mount NFS via Disk Utility)

Hi R.G.,

I don’t regard myself as a UNIX/ZFS expert at all, but I’m a self-taught user of Solaris & ZFS, and have found that it does what I need very well, seems to be the best current storage solution, and it works — oh yes, and it’s free to those willing to learn and spend the time required to learn how to use it.

I can’t say whether your proposed hardware will definitely work or not, as I’m not familiar with the hardware.

You can check for your hardware on the Solaris HCL (hardware compatibility list).

However, 6 drives should be good for a data pool comprising capacity of 4 for data and 2 for parity (one RAID-Z2 vdev).

And 2 mirrored drives for a boot pool is also a good idea.

In general, Supermicro make excellent motherboards, and Intel chipsets are well-supported by Solaris.

As well as the HCL, try Googling for the various components including motherboard and solaris and see if there are any happy/sad users.

Just to answer directly your questions:
1. Yes, 2 pools are a good idea: (1) 2-drive mirror boot pool, and (2) multi-drive data pool using a single RAID-Z2 vdev for strong redundancy protection and simplicity of maintenance.
2. Six SATA ports for a RAID-Z2 vdev for your data pool are fine — see 1.
3. You need to verify Solaris has a driver for that graphics hardware — check the HCL or Google.
4. In general Intel ICHxx chipsets are well-supported by Solaris, as are their ethernet cards, although I have no personal experience with them, having gone for an AMD/NVidia setup with ECC memory.

Also, I would strongly recommend you use ECC memory, as ZFS is all about data integrity, and I don’t think you want to write garbage, due to corrupted flipped bits in memory, to disk.

If any of the ZFS stuff on this site helps you to get a good working ZFS system up and running soon, then feel free to thank me by clicking on a “Buy me a beer” button I shall add soon 🙂

Good luck!

Cheers,
Simon

For setting permissions / ACLs on new file systems, via SSH or VNC on your client machine, you can use chown to change owner, and use ‘zfs set’ to change the ACL properties — e.g. something like:

# zfs set aclinherit=passthrough tank/home/fred/photo
# zfs set aclmode=passthrough tank/home/fred/photo
# chmod A=\
owner@:rwxpdDaARWcCos:fd-----:allow,\
group@:rwxpdDaARWcCos:fd-----:allow,\
everyone@:rwxpdDaARWcCos:fd-----:deny \
/tank/home/fred/photo
# chown fred:fred /tank/home/fred/photo
# zfs set sharesmb=name=photo tank/home/fred/photo

Hi Shaky,

On my Mac, I have setup autofs to mount my NAS CIFS shares using the following method:

Add the following line to /etc/auto_master :

/- auto_direct

Then create or edit /etc/auto_direct and include the following line for each share you wish to mount :

# photo library
/Users/simon/nas/photo/photo -fstype=smbfs ://simon:password@192.168.1.3/photo

In my home directory (/Users/simon) on the Mac, I created a directory called ‘nas’ and within that I created the ‘photo’ directory. This ‘photo’ directory is where autofs will create a mountpoint called ‘photo’.

I could have put all the mountpoints directly into the ‘nas’ directory, but this had the nasty effect that any access to the /Users/simon/nas directory via the Finder caused the Mac’s autofs to mount all CIFS shares listed in /etc/auto_direct, resulting in an annoyingly long delay before the Finder became responsive. The same problem occurred when opening or saving a file to the NAS from any application via the Finder open/save dialog window. Thus to eliminate this problem, I mounted each of the shares in separate individual directories, each one being one level below the ‘nas’ directory.

Later on, I had the idea to force autofs to mount all the shares when I logon to my Mac user account. I have a script which runs when I logon to my Mac user account. I called the script ‘mount_shares’ and it’s something like this:

#!/bin/bash

printf "\n" >> /Users/simon/log/mount.log
date >> /Users/simon/log/mount.log

printf "\nAttempt to mount photo:\n" >> /Users/simon/log/mount.log
ls -l /Users/simon/nas/photo/photo >> /Users/simon/log/mount.log

The ‘ls’ causes autofs to mount the share. Add a line like this to your ‘mount_shares’ script and then make the script run automatically when you login by adding the script to the ‘Login items’ of your user account in the ‘Accounts’ section of System Preferences.

Finally, in order to ensure the autofs mounts stay mounted longer than the default 10 minutes, edit the /etc/autofs.conf and modify the timeout to something like this:

Macintosh:etc simon$ cat autofs.conf
#
# This file is used to configure the automounter
#

# The number of seconds after which an automounted file system will
# be unmounted if it hasn't been referred to within that period of
# time.  The default is 10 minutes (600 seconds).
# This is equivalent to the -t option in automount(8).
AUTOMOUNT_TIMEOUT=360000

This causes autofs to mount the shares for 360000 seconds, or 100 hours.

Hope this helps.

Cheers,
Simon

Hey everyone,

I am slogging it out with my Solaris Express 11 box connected to a Windows 7 machine via Infiniband (drool). The storage pool on the Solaris machine has 3 X 2tb Deskstar 7K3000, I 5gb log and 30 gb cache (cache and log are both on a sandforce ssd). Reads are incredibly fast and over much more that 150MB/s but writes are depressingly slow at 30mb/s (max). I have tried disabling de-dupe (no luck) and setting log priority to through put but neither of these settings help much.

When I attempt a large write it starts of being quite speedy (100mb+) then drops off after a while (<10gb).

The system is running an i3 cpu with 8gb of ram. Running top show very little CPU usage (~20%), all but 800mb worth of ram but no swap.

Any advice would be greatly appreciated.

Kind Regards

Jim

It has been a couple years since anyone has posted but I came across this blog entry and would like to thank the author for the great tutorial. I am just learning but plan to put together a NAS server with zfs within a few months. I’m thinking of using openindiana with 3 red wd 3tb drives in raidz1 and a second usb external pool. I will be accessing the shares with windows, Mac, and android machines. Mounting Cifs seems to be difficult on Android 4.2,not sure the best way to go.

Any new developments or suggestions?