SAN Redundancy and Round Robin Multipathing

It's just a network
Yes, the N in SAN stands for network. Ethernet networks are not loseless. In a Fabric Channel capable network, you can't loose packets. You just lost data. And when a server thinks the disk has lost data, bad complex things happen. Packet loss in an ethernet network is expected. In a FC SAN, it's bad.

Queue Depth
Queue Depth is a term any knowledgable SAN/Storage administrator knows, but surprisingly often has no idea how the value actually works. Queue Depth is a simple concept, it's basically how many I/O's the HBA will throw onto a SAN before queuing. People often times think (and somewhat logically) that if you increase the depth of your queue then more things will queue, but the truth is this:

The larger your queue depth the more I/O's that go to the SAN before being queued on the server.

Thus HBA queue depth is one of the main parameters you can tune when dealing with storage performance issues on a server attached to a SAN

Finding the correct setting usually requires a bit of testing, sometimes a little luck, and a lot of dependance on your vendor:

HP: Here's a white paper
HDS: Here's a formula
EMC: Here's a quote
Oracle: OMG YOU USE A SAN AND THE EVIL RAID 5....BURN THE HERETIC BURN THE HERETIC
NetApp: Please ignore the effects of WAFL on reads after everything becomes a random write

All kidding aside, each vendor has best practices and ways of analyzing what queue depths you can push to.

Mulitpathing
Let's take a further step into the server. The I/O's have to get to the HBAs, and sitting in that I/O stack is some multipathing software that helps keep all of the HBAs busy

I'm going to stick to three major types of multipathing software:

• Full featured
• Queue Depth Monitoring
• Round Robin

Full featured multipathing is the software that really, honestly, knows how badly you designed your SAN. It knows that you skimped on the cable for that link between those switches. If it's the same vendor as your storage, it also probably knows a little about how the host ports on the storage array are doing. In other words, this is the software that gets huge performance gains because in all likelihood, it's smarter than you. You are going to pay for this intelligence, and it's probably not going to be cheap. But if you have a lot of servers or have some ones that require high performance, this may save you from reoccurring issues.

Look at your server as a city with four highways that all lead to the beach. At all times, the city knows which highways are busiest and will dynamically adjust as the four highways become faster and slower.

Queue Depth Monitoring is multipath software which understands that if one of it's paths is slow, that the HBA is queueing up and that it should use that path less. It has some of the benefits of a full featured multipath solution in that it does respond positively in cases where there is congestion, but that response will always be slower. The more capable OS based multipath solutions include this.

Think of that city again. If one highway clogs up then eventually the highway will stop getting used for some period of time, allowing it to clear.

Round Robin is the final multipath software I will discuss. I include this because if you use Red Hat native or VMWare* esx/esxi, then you at best have round robin multipathing. Basically, each path is used in order before moving to the next path, then repeating from start. Congestion is completely ignored, except of course in the error log where the server complains the storage is too slow (0x2 for you VM folks)

*A special note for you VMWare folks, by default VMWare's round robing sends an almost criminal number of I/Os down a path before using additional paths, leading to a situation where round robin is basically active/passive.

Backups
Did you buy a whole lot of fancy deduplicating VTLs that are emulating Storagetek Tape Library's and you stuck them all on the same Fabric that is shared with your production data?

You just put a mismatch of data on one half of your redundant SAN.

Ok, so what?
Well, your Red Hat engineer who insisted that you use native Red Hat MPIO because to use third party mulitpathing you have to do kernel recompiles for upgrades has now placed into service on your SAN servers that will not understand that yes, half your fabric is congested. It's going to happilly continue to throw data down that path, possibly find enough congestion to timeout and then, because Red Hat protects itself by making the volume read only, have a unusable system/application.

If you want a redundant SAN environment, round robin multipathing does not cut it
This is the issue. People design their SAN to be redundant. They put multiple HBAs in each server. The two fabrics in the SAN are there so you can break one. But the moment you use round robing multipathing you have killed your redundancy with regards to things such as congestion, packet loss, bad cables, and other ways that make paths not ideal. The only way round robin stops using a path is if it goes down. When you choose round robin, you are choosing to ignore all the engineering that went into designing your SAN.