Thanks for the compliments. Unfortunately, no I’ve not had a chance to try out the eSATA port on the back of this Asus M2N-SLI Deluxe motherboard. I’ve seen other people say that the SATA port mounted on the mobo inside the case that is also driven by the JMicron JMB363 chipset is not supported yet in OpenSolaris, although I’ve personally not tried to get it working yet, as with 6 SATA ports already available, I’ve not had a need to. If you do discover that the JMicron eSATA port is in fact supported, I’d appreciate it a lot if you could let me know. I did see a nice drive enclosure for a single SATA disk powered via an eSATA port: the Antec MX-1 which is only about 50 euros/dollars, excluding the drive. See here: http://www.silentpcreview.com/article728-page1.html

Good luck!

Simon

Many thanks for all the great info on this series of posts. I just bought an Ultra 20 case off ebay for almost nothing and plan to fill it with this or a very similar mobo. Have you had a chance to test if the eSATA port works ok? It looks like the JMicron 363 chipset is supported from b82 onwards. Trying to hunt down a suitable cpu now…

Cheers z0mbix

See the following article for more details: http://blogs.sun.com/mhaywood/entry/powernow_for_solaris

To support CPU frequency scaling with AMD processors, it seems the processor must be family 16 (10h) or greater, and of course, mine is family 15.

However, family 16 is the Barcelona processors, and these seem to use between 80 and 120 Watts, according to this page: http://www.theinquirer.net/en/inquirer/news/2007/09/10/average-power-consumption-is-anything-but

So my 45 Watts doesn’t sound too bad after all.

Here’s the diagnostics for my processor (AMD Athlon X2 BE-2350 45W TDP):

bash-3.2$ kstat -m cpu_info -i 0 -s implementation
module: cpu_info instance: 0
name: cpu_info0 class: misc
implementation x86 (chipid 0×0 AuthenticAMD 60FB2 family 15 model 107 step 2 clock 2110 MHz)

bash-3.2$ kstat -m cpu_info -i 0 -s supported_frequencies_Hz
module: cpu_info instance: 0
name: cpu_info0 class: misc
supported_frequencies_Hz 2109683672

As you can see, it only supports one frequency: 2.1 GHz.

If you are concerned about high power wattage, then maybe you could even things out by trying ultra thin clients SunRay as I have posted here on “Drive temps”? They each use 4 watt, and if you connect a few of those to the server, you can ditch all your other computers. Now you have your server using wattage, and your desktop computer using wattage, your wife’s computer uses power, etc, summing up to several 100 watt. If you try SunRay instead, then only your server uses power and you can have 10 users simultaneously, each consuming 10% of one server’s power.

Yes, it’s funny we chose similar hardware. Which make/model PCI Express SATA card did you use (Silicon Image 3132-based) ?

I like your pool name — blackhole That sure is quite a beast with 8 drives! And you’re right — when these 1TB drives get a lot cheaper, it will be terrific.

I’d be interested to hear how your efforts to reduce power usage go. Yes, I heard other people mention that the JMicron SATA connector seems to be unsupported so far within OpenSolaris.

I didn’t have lockups with the Gnome desktop, but I had strange cases where lots of spurious mouse events would appear, causing the Gnome interface to be almost impossible to use, but I think this might be caused by the 2-port Linksys KVM switch that I’m using to switch between 2 Solaris boxes (fileserver and backup server).

Thanks for the best wishes with this setup — I’ve got quite a few more ZFS articles I plan to write when I get a bit more time…

Good luck with your system too!

Enjoy the music at KOKO — which band? Not sure yet of my plans for October.

Oh, I almost forgot. When I did use power management on the disks, I remember it did take a while (30 seconds sounds possible) for the data to come back online. I think you are right — there is a staggered spinup of the disks, so that each disk comes back online one at a time. Although this might be inconvenient, this is probably a good idea, as the disks consume a lot more power when they are spinning up, and if all eight drives, in your system, were to spinup at the same time, the power requirements might exceed the capabilities of the computer’s power supply, in some cases.

# zpool status
pool: blackhole
state: ONLINE
scrub: none requested
config:

NAME STATE READ WRITE CKSUM
blackhole ONLINE 0 0 0
raidz1 ONLINE 0 0 0
c1t1d0 ONLINE 0 0 0
c2t0d0 ONLINE 0 0 0
c2t1d0 ONLINE 0 0 0
c3t0d0 ONLINE 0 0 0
c4t0d0 ONLINE 0 0 0
c4t1d0 ONLINE 0 0 0
c5t0d0 ONLINE 0 0 0
c5t1d0 ONLINE 0 0 0

errors: No known data errors
# zfs list
NAME USED AVAIL REFER MOUNTPOINT
blackhole 742G 2.38T 742G /blackhole

Just imagine migrating this thing to 8×1TB drives… just wait until these drop to 50 euros or so… mmm…

The beast draws about 160w with all disks on, and it dips to around 102w when the array goes idle (1 hour). I think I might shave another 20w or so by removing the video card and going headless.
Maybe I could get even lover by replacing one of the controllers with a port multiplier. Will lety you know when I do this.

About the onboard JMicron, I think at the moment it’s still not supported: I tried bot using if for boot and for hooking up a secondary drive, but had no luck at all. Maybe one day we’ll be able to use this lonely port, too…
Speaking of problems, I experienced frequent lockups with the Gnome desktop. Switched to XFCE and the problems disappeared. Did you find similar problems?

Well, I’ve talked far too long.
Best wishes for the further developments of your rig, and I’ll wait to hearing from you soon!
You found a fellw reader for your blog. Who knows if we’ll meet one day…
I’ll be in London in late october with my wife, to attend to a concert at the KOKO. Who knows…

Great write up

Once I get time to get that setup script together, I will post it here.

Yes, having the graphics card is useful. Some apps don’t run headless anyway. And if you get a low-power, passively-cooled graphics card like the Asus EN6200LE (30 euros) then it uses very little power and makes zero noise.

I would be interested in your setup script if thats in the public domain

I’ve decided just to shell out the £30 for a cheap PCI-E graphics card, I was desperate to get this server up and running, plus it never hurts to have graphics in place should something go horribly wrong down the line.

Thanks

I read somewhere that non-server motherboards don’t allow successful POST running headless, but I didn’t verify that categorically is the case. One possibility to debug this problem could be to enable POST error logging, then remove video card and reset the machine. When it fails to POST, shutdown, add the video card, power up and inspect the BIOS error logs for any useful log info as to why it failed to POST. If you get anywhere with this, I’d love to hear your findings.

Regarding usage of Nevada or Indiana (aka 2008.05), see a comment I made on the OpenSolaris forum here:
http://opensolaris.org/jive/message.jspa?messageID=235112#235119

Basically, if you use Indiana/2008.05, unless you fork out the $320+ you won’t be able to use IPS to update your system (like the apt-get in Debian Linux), so for me, it is a bit useless. I would rather use Nevada for now and have freedom to update my system if I see a need to. On my todo list is to create a script that will recreate my environment on a new build of Nevada after a new version install.

Other than that it all seems to be going well, just having a conversation on the osol-discuss mailing list about whether to stick with a nevada build or go over to the main opensolaris distro

PS. If someone adds a vdev to grow your ZFSraid, make sure you dont type wrong. Someone typed wrong and added a single drive to a ZFS raid, and then he got new ZFS raid = old ZFS raid + 1 drive.

To experiment with ZFS, you can create UFS files and format them as ZFS. Then you can safely experiment with ZFS files, acting as file systems.
http://www.cuddletech.com/blog/pivot/entry.php?id=446

The reason I ask is that when I was using a single gigabit ethernet connection between Mac Pro and the ZFS fileserver, using CIFS sharing I seem to recall getting speeds around 40 MBytes/sec, and when I pumped it up to using 2 gigabit NICs with Link Aggregation, I started getting transfer speeds around 80 MBytes/sec, and this is using the cheap Asus M2N-SLI Deluxe motherboard and its built-in six SATA connectors.

I like your interest in getting a low-power solution going. Using the hardware listed above in this article, my system draws around 120W. So far, I didn’t get success with low power modes, as the suspend (S3) seemed to work, but resume did not work and I did not have the required information to identify which driver was unable to resume correctly.

Also, I like the idea of using Solaris Zones to run Windows, and accessing the Windows session via a Sun Ray. At some point in the future I might try that out.

When you get your system running, it will be interesting to see your power usage readings.

My server is my workstation. To the server/workstation I will connect a 30″ monitor and a Nvidia GeForce. I will dual boot when playing Windows games. I will shut off the power to the ZFSraid most of the time, it will mainly be used for storage. Therefore, I will have a >250GB system drive that holds temporarily stuff: movies, MP3, etc. Every know and then, I will copy stuff from my ZFSraid to the system drive. As soon I have something valuable on the system drive, it will get copied to the ZFSraid. Later the system drive will be upgraded to SSD drive with 0.5 Watt. I dont need access to all my Movies and documents all the time, Ive found out. I just copy a boatload of things that I want, to the system drive and >250GB is enough for a long time, if you delete Movies, etc, regularly.

A standard hard drive uses ~10 Watt. My ZFSraid uses 40 Watt. With “zpool export” and “zpool import” I can turn off the power to the drives at will. I dont know if the SATA card supports hot swapping, but it doesnt matter. I just reboot. I dont know how I will turn off the power to the drives. I am thinking of cutting each power cables to the drives and connecting a power on/off switch. What makes most sound in a computer, is the hard drives, Ive found out.

A quad core Penryn 9450 use ~7 Watt in idle, going up to ~50 Watt under full load.

A GeForce Nvidia 9800 uses 200 Watt full load, and like 60 Watt idle. However, there are new chipsets from Intel soon, with similar functionality to AMD 780. These next generation chipsets allow to turn off power to the external graphic card, and only use the internal. This way, the Nvida 9800 is only used when playing games. If we look at AMD 780, it looks that 30″ monitors are not supported with the internal 780 graphic card. Neither is Windows XP allowed, only Vista is allowed. And Vista is less optimal. I hope Intel future chipsets will fix these two problems. I also wonder how long it will take before Solaris supports these two functions. But Nvidia has good Solaris support with new drivers all the time.

To the server I will connect my SunRays and place them throughout the building. (BTW, I am the one who told you about SunRays). The server will be in the basement. I will sit with the 30″ in the basement. When someone surf/MP3/Movie/works they can use the SunRays. The server just send bitmaps to the SunRays, that means using SunRays will not be any slower than if you used the Server directly. There will be no difference, except graphic bandwidth - movie playback. But compilations, computations, working, etc will not be any difference at all. The SunRay servers normally doesnt have any graphic card at all. It generates the bitmaps in RAM. BTW, VirtualBox 1.6 has just been released, allowing to run VB from within Zones in Solaris. These virtual zones are very lightweight, and I can give root access to a zone. Within a zone, no one can break out into the global zone. So, people can use Windows XP from SunRay. Or you can use Windows + Solaris + SunRay:
http://www.youtube.com/watch?v=BCNWB_slB-Y

So, I will have a 9450 quad core Penryn that uses ~10 Watt idle. 4-8GB RAM uses 10 Watt each memory stick. One 250GB System drive ~10 Watt idle. The ZFSraid is normally shut off. The graphic card is normally shut off too. And I will later change the 250GB system drive to SSD using 0.5 Watt. Each SunRay uses 4 Watt, I can have a boatload of them in a drawer. This is my master plan. Opinions? (Al Gores movie influenced me a lot. Save energy whenever you can!).

BTW, Correction: the AMD 940 mobos often has PCI-X slots, NOT the 939 mobos. So a cheap solution would be a 940 mobo + dual core opteron + AOC SATA 2 card PCI-X. If you can edit my earlier comment, please change 939 to 940?

ZFS works best with 64-bit processors, so although 32-bit processors work, they are far from optimal.

Fast read/write speeds are cool, as long as you have matching power to transfer the data across the network to connected machines. How fast is your network in real-life sustained large capacity transfers with the system you mention?

Using ECC makes a lot of sense on something like a storage server where you want data integrity.

The drives show up as c0t0d0s0 etc.

The similarity between the Asus M2N-E and M2N-SLI Deluxe motherboards is interesting to know. Unless you need the eSATA, SLI, extra SATA port or extra gigabit ethernet connector and the additional items you mention, then the M2N-E looks to be a good board for AMD processors (AM2 socket).

I run with SXCE, as I want the ability to benefit from regular bug fixes, without having to pay $300+ annually that Sun wants to allow you to use the IPS pkg command to update your system packages, like the old ‘apt-get update’ used to work in Debian Linux.

Looks like you have some nice hardware components there — especially the Supermicro SATA card. I think I might use the same card one day when I build a new system.

Ive heard that it is the same chipset as in the SUN X4500 Thumper machine with 48 drives. The card gets detected automatically by Solaris, during install. The card is PCI-X (not PCI-express). PCI-X slot can only found on server mobos. If you plug this card into an ordinary PCI (which I have done), the transfer rate will drop.

A normal PCI at 32bit/33MHz has a peak transfer rate at about 133MB/sec. Which is not far from SATA II 150MB/sec, but it would be nice if I could get a full 150MB/sec - but it wont happen with this card, unless it is either 64bit or 66Mhz - then the transfer rate will increase to 267MB/sec
http://en.wikipedia.org/wiki/List_of_device_bandwidths
If used correctly as a PCI-X card, the max theoretical bandwidth with this card is 1.07GB/sec, which should suffice for most people’s need.

The manual claims the card is 64bit 133MHz PCI-X. http://www.supermicro.com/products/accessories/addon/AoC-SAT2-MV8.cfm

I go for an external SATA II card, that way I dont have to worry about picking the right non-server motherboard. If I want an external card, I have to resort to PCI or PCI-express on plain mobos. I dont know of SATA PCIe cards, so I guess the only alternative I have left is PCI cards. This card will follow me between my upgrades, from P4 now, to Penryn.

Everything works flawless with the card.

There is one problem with transfer rates: I have a P4@2.4GHz and 1GB RAM. I get like 20 MB/sec. That is due to the P4 being a 32 bit CPU. As ZFS is 128 bit, it doesnt like 32 bit. With 64 bit CPU you get transfer rates in excess of >100MB/sec. I will upgrade my Solaris machine to a Penryn Quad core. Nothing needs to be changed for ZFS running on 32 bit CPU, when upgrading to 64 bit CPU. Everything will function automatically.

I’ve heard that an old Opteron 940 mobo, often has PCI-X slots. And the CPU is dirt cheap. And the mobo to. And 3200 memory too. Add this AOC Micro card, and you got one hefty server capable of 1GB/sec transfer rates.

Regarding ECC Memory, Solaris has something called “self-healing” which means it is kind of ZFS-sque checksums for RAM. It means, RAM will be checked against errors. (I think, please google on self healing). With ECC, makes Solaris doubly secure against RAM failures.

The M2N-E looks to be the exact same mobo as the M2N-SLI Deluxe. If you lift the M2N-E model sticker on the mobo “M2N-SLI Deluxe” is silk-screened underneath! It looks like they just pulled some components (2nd gigE, fiber audio, 2nd x16 PCIe slot). Comments?

I built this rig for a home server and it’s rock solid running Solaris 10 U5 and SAMBA:
M/B: Supermicro H8SSL-i2
CPU: AMD Athlon 64 X2 5600+ dual-core 2.793GHz
MEM: 4GB (2x Kingston KVR800D2E5K2/2G 2GB Kit DDR2-800 PC2-6400 ECC)
I/O: one PCI-X 64-bit/133MHz slot, two PCI 32-bit/33MHz slots
VID: Integrated ATI ES1000 Graphics Controller (8MB)
DISK CONTROLLER: Supermicro AOC-SAT2-MV8 133MHz PCI-X SATA2 controller
DISKS: 6 x Seagate ST3500630AS SATA/300 500GB disks (7200 RPM, 16MB cache)

I was going to run SXDE 01/2008, but SXDE is dead now. SXCE is too bleeding edge as is OpenSolaris 2008.05 (using ZFS as root is very cool, but I want more stability for my NAS). I may run OS 2008.05 on the new box tho.

Configured 2 disks as root and mirror, 4 disks in a RAIDZ pool. Disks show up as SCSI (c0t0d0). Drives are currently factory jumpered for 150MBs. Haven’t tried un-jumpering them for 300MBs, but it probably wouldn’t make much difference. The OS notices tho:
May 19 18:47:52 toybox sata: [ID 514995 kern.info] SATA Gen1 signaling speed (1.5Gbps)