Cheers
Otto

-James

Before I begin I would like to say a bit on my background. I use to work for Netgear as tier two senior support engineer for their prosafe product line which includes their managed switches. After that I worked for Juniper Networks one their ERX support team. So I have a bit of background on this subject.

Lets cover terminology for managed/smart switches.

1) A trunked port is a port between two switches, it does not have to more than one port. This is where we get the word ‘trunk,’ like the a tree trunk, because switches form the trunk of a network and this is the port that the switches communicate over. Because switches are the trunk of a network these trunk ports generally need to have more bandwidth as the port will be aggregating all the inter-switch traffic. Sometimes venders will provide special high bandwidth ports, such as GigE on a Fast switch, or 10GigE on a Gig switch, or even proprietary back-plain designs. However, sometimes they do not, and on a managed switch that supports 802.3ae you are able to LAG multiple ports together to increase inter-switch bandwidth. This is probably where the misconception of the term ‘trunk’ came from. This is important to understand as on some switches trunk ports communicate special command information that allows multiple managed switches to operate as one large switch matrix, and these trunk ports are not allowed for general switching.

2) The term for aggregated ports is called a LAG, Link Aggregation Group. This was true when I was supporting $6k switches at Netgear and was still true with I was supporting million dollar routers at Juniper.

On to 802.3ae…

802.3ae does not define a load balancing method; that is up to the vender.

At Netgear we had two classes or ‘non-dumb’ switches, smart switch and managed switches. The funny thing is that the smart switches, which the above mentioned GS108T (G is for Gigabit, S is for smart, 8 means there are 8 ports), are not all that smart. They way dumbed down managed switches with less hardware. They feature things like only webgui operation, and reduced feature sets, and reduced granularity for the features they do support. For instance the smart switches sports IP based round robin aggregated port packet distribution. Once a port has been assigned to a given IP it will always get that IP until the aggregation table is flushed. All of Netgears switches that start with two letter, the second being an S, are smart switches.

The actual managed switches are the ones with three letters followed by four numbers. Examples would be GSM7324 and GSM7224. In this example these would be ‘G’ for Gigabit, S for Switch, M for managed, the second number is the layer (2 is a layer 2 switch, 3 is a layer 3 switch), and the last numbers are the port count. Aside for the greatly increased feature set, much improved granularity, addition of CLI control, very low latencies, one of the managed switch’s claims to fame is it’s hash based packet load balancing. (This is proprietary design as the 802.3ad spec does not specifically address load balancing.) As packets traverse the switch their frames are hashed and tracked in a hash table, this allows the switch to insure in sequence layer two communication, however the higher layers (TCP/IP can handle out of order processing as there is no guarantee that packets traversing large networks, thing the Internet, are going to arrive in the same order they were sent) are then load balanced on the ports based on Netgear’s proprietary algorithm. This allows two hosts, for instance, a Mac Pro and a Solaris server to achieve 2Gbps speeds. However you do have to have 802.3ae configured on the Mac, the switch ports facing the Mac, and the Solaris box, and the switch ports facing the Solaris box.

Concerning bandwidth over a network. Yes, you can see full Gigabit speeds over the a network assuming the following.

1) Your switch needs to be able to actually handle it. Just because you have a “Gigabit switch” doesn’t mean that the switch fabric can handle Gigabit rates to all ports at the same time. If you have 8 ports, and they are full duplex, then you have 1Gbps * 8 * 2 (1Gpbs for each direction) which would mean the switch fabric would need to be able to handle 16Gbps! Think about a 24 port or a 48 port…

2) Your cable needs to be able to handle the speeds. Things that are going to affect this is impedance, signal noise and distance. Make sure you are running CAT6 cable. If you are distance shouldn’t be much of an issue in home implementations. Be mindful of things that will generate noise, lights, electric motors, speakers, etc… don’t run your cable by these. If you have to cross an electrical cable do it at a 90.

3) You have to remember that there is protocol overhead that has to be accounted for. Gigabit = 1024Mbps = 128 MBps. SMB, AFP, iSCSI, are all TCP protocols which means their is a 3 way handshake. (SYN, SYN-ACK, ACK) So there is allot of over head packets for each communication. More over there is over head for each packet, you have the layer two frame, the IP header, the TCP header, the SMB/AFP/iSCSI header, and then finally the data. This is why jumbo packets are recommend for large data throughput networks, it lowers the header vs data ratio. However this doesn’t affect TCP’s inherent overhead or the fact that it prefers reliability over speed. For instance if your network becomes saturated and an ACK or SYN-ACK does not come within the expected response time TCP down scales its window to easy network burden and to increase the chances that it’s data will get through. Remember TCP was created back in the late 60s early 70s with UNIX and C and networking and when the networks weren’t very reliable.

4) IEEE 802.3ab, Gigabit over UTP, compatibility is not enough to deliver Gigabit rates, the hardware has to be able to keep up with line speeds and the driver has to be able to keep up with the buffers. The chipset and the drive makes are HUGE difference as to what the speeds will be. I personally have had good luck with Intel, and I have seen many issues with integrated chipsets.

In my testing with my Mac Pro and my Macbook Pro over a DLinkDGS-2208 with CAT6 cable I have achieved sustained speeds of 87MBps. I did this with setting both of the Macs to full duplex with jumbo frames. Then I created a ram disk on both machines and copied a 2GB disk image back and forth across the network from one machine’s ramdisk to the other’s. This isolated the network component of the equation. The protocol I used was AFP running on top of 10.6.2 on both computers.

Good website.

I’m currently using a Solaris server with 2 aggregated 1Gbps links with a switch that supports trunking (but not LACP), and I can push the network traffic on the server up to 195-205MB/s with 2 clients, and up to 120MB/s with a single client. The clients also have dual aggregated 1Gbps links.

First, I configured the switch:

Switch# configure terminal
Switch(config)# interface range gigabitethernet0/17 -18
Switch(config-if-range)# spanning-tree portfast
Switch(config-if-range)# switchport mode access
Switch(config-if-range)# switchport access vlan 10
Switch(config-if-range)# channel-group 5 mode active
Switch(config-if-range)# end

Then I configured Open Solaris snv_97

# svcadm disable /network/physical:nwam
# ifconfig e1000g0 unplumb
# ifconfig e1000g1 unplumb
# dladm create-aggr -d e1000g0 -d e1000g1 1
# ifconfig aggr1 plumb
# ifconfig aggr1 10.1.1.3 netmask 255.255.255.224 up
# svcadm enable /network/physical:default
# dladm show-dev
# dladm show-link
# dladm show-aggr
# ifconfig aggr1

Unfortunately, this didn’t work. On the Cisco, the ports were ORANGE and I got the following log messages:

“LACP currently not enabled on the remote port”

Ok. So the ports are set to ACTIVE on the Cisco side, but not on the Solaris side. So I had to modify the aggregate interface with this command:

# dladm modify-aggr -L active aggr1

Now it works. Of course, I learned the 802.3ad standard does not allow round robin’ing of packets. I need to figure out a way to load balance the two interfaces. The Cisco supports the following:

Switch(config)#port-channel load-balance ?
dst-ip Dst IP Addr
dst-mac Dst Mac Addr
src-dst-ip Src XOR Dst IP Addr
src-dst-mac Src XOR Dst Mac Addr
src-ip Src IP Addr
src-mac Src Mac Addr

You’re welcome — it’s nice to be able to help others.
Using CIFS sharing, with one GbE link I got around 40 MBytes/sec, and with trunking enabled using two GbE links I’m getting around 80 MBytes/sec.

About your trunking efforts:
I’m not surprised that having link aggregation at one end (solaris), but not the other end (switch) of your link appears to work. Each transmitted frame is an atomic operation on each end of the link: The sender does what he will, and the receiver tries to make sense of it. My bet is that Solaris was trunking, and that the MAC address table on your switch was being constantly thrashed.

When you get trunking fully supported, I don’t think you’re going to see benefits between a single pair of computers. The trunking mechanisms go to great lengths to not allow frames within a flow to be mis-ordered. A flow is usually defined by the 5-tuple: src_ip, dst_ip, protocol, src_port, dst_port. …But the particulars are configurable. Frame mis-ordering is accomplished by assigning a flow to a particular link, and making sure that all frames in the flow are sent over the same link.

Note that the link chosen by the switch doesn’t have to match the link chosen by the server because they balance the links independently.

When your fileserver is performing bulk transfers to or from multiple systems you might see a throughput benefit. If not, then the benefit is limited to link redundancy.

/chris

Thanks for the info — I think I saw the other day that one of the links on the Mac or Solaris box didn’t appear to have flow control enabled, so I will take a closer look at that. Cheers.

Also, around the same kind of price, I’ve seen a Linksys (now Cisco) switch: the SLM2008, which also appears to support IEEE 802.3ad for Link Aggregation. I don’t know how these 2 switches compare with each other, but I might take a read.

I see also there is a Linksys SRW2008 switch, which seems about twice the price.

If I really want to checkout Link Aggregation then I’ll need to get something like one of these switches that supports IEEE 802.3ad.

For now, I’ve destroyed the aggregated link and returned to NWAM with a single Gigabit ethernet link.

Now I’ve seen aggregation though, I am tempted to get one of those IEEE 802.3ad compliant switches as they’re quite inexpensive for a prosumer model like the first two mentioned above — something like $100 / €100.

I too have two Gb ethernet ports and have looked at aggregating them. The switch I’m considering is the Netgear G108T which is a smart switch and claims to support link aggregation. For the prosumer it looks to be at the correct price point – unless you can get hold on a Cisco at a good price that is!

with both ethernet cables attached to the switch:
# dladm show-link
LINK CLASS MTU STATE OVER
nge0 phys 1500 up –
nge1 phys 1500 up –
aggr1 aggr 1500 up nge0 nge1

now disconnect nge0 cable:
# dladm show-link
LINK CLASS MTU STATE OVER
nge0 phys 1500 down –
nge1 phys 1500 up –
aggr1 aggr 1500 up nge0 nge1

I still have a usable connection (I can surf), so it’s now using nge1 — or the ether

now plug nge0 cable back in and pull out nge1 cable from the switch:
# dladm show-link
LINK CLASS MTU STATE OVER
nge0 phys 1500 up –
nge1 phys 1500 down –
aggr1 aggr 1500 up nge0 nge1

I can still surf, so it’s using nge0.

So the redundancy/fail-over aspect of the aggregated link is working.

When I think of a heavy enough test to really stress the link I will see if it’s using both links or not — will probably need some network monitoring tool to to see the figures…

It has separate entries for 802.3x and 802.3ad:
* 802.3x: Full Duplex and flow control; also incorporates DIX framing, so there’s no longer a DIX/802.3 split
* 802.3ad: Link aggregation for parallel links

From this, it seems that my switch doesn’t support link aggregation, as it only mentions “802.3x Flow control” in the specs and not “802.3ad Link aggregation”. Oh well, never mind, aggregation was never a requirement when buying the switch, just something that occurred to me later. But it’s interesting to know these things for possible future projects.

But my question is, does link aggregation always rely on the hardware — i.e. the switch, or can it be done in software using the OS, in this case Solaris?

From these pages, it seems like the switch definitely needs to support link aggregation:
http://blogs.sun.com/droux/entry/link_aggregation_vs_ip_multipathing
http://blogs.sun.com/droux/entry/link_aggregation_plumbing

I have a DLink DGS-1008D green ethernet 8-port Gigabit switch – see here:
ftp://ftp.dlink.eu/datasheets/DGS-1008D.pdf

Of relevance, perhaps, in this data sheet it says it supports “IEEE 802.3x Flow control”, which I assume covers IEEE 802.3ad, which appears to be what’s required for link aggregation / port trunking:
http://en.wikipedia.org/wiki/Link_aggregation

Does this look like this switch should support link aggregation to you?

Here’s additional info from their website:

* 8 10/100/1000 Mbps Gigabit ports on Cat. 5
* 16Gbps switching fabric
* Auto MDI/MDIX cross over for all ports
* Secure store-and-forward switching scheme
* Full/half-duplex for Ethernet/Fast Ethernet speeds
* Blazing 2000Mbps full duplex for Gigabit speed
* IEEE 802.3x Flow Control
* Plug-and-play installation
* Easily installable on desktop