Call me slow, but I’ve only recently come to understand what the Xserve RAID really is, and it’s not just a RAID.
Before my revelation, I had figured that, out of the box, the Xserve RAID was strictly a DAS RAID device, connecting via Fibre Channel to a single Xserve, much like the SCSI RAID boxes we already have connected to servers of several stripes in our server room. Yes, the Xserve RAID can work this way, but as I found, it’s not the only way. I also confess that though I was aware of Apple’s Xsan software, I thought that only with it could you turn the Xserve RAID into a true multi-hosted SAN.
It’s not that Apple doesn’t clarify this on their site, but I never looked into the details until it was time for us to purchase a SAN solution. Now that I have looked, I would suggest calling the Xserve RAID the “Xserve SAN” or even the “Apple SAN”, since out of the box that’s what it is, whether you run it with Xsan or not.
So, not only will the Xserve RAID work as a DAS, but you can also plug one into a Fibre Channel switch, and its volumes will be available as local storage to the servers on that switch (none of which have to be Xserves), making it a true SAN device.
What Xsan adds to this, then, is the means for multiple servers to access the same Xserve RAID volumes concurrently. This is something valuable in more collaborative environments like video production, but it’s nothing we would need for hosting our mail, file, or data stores that are accessed only by their respective servers.
This turned out to be a big deal since even when I considered an Xsan system, which requires two Xserve boxes to run, its price was still about half that of other FC or iSCSI solutions I was looking at. Not bad, but now that I know I can have an Apple SAN without Xsan, I’m looking at a price of roughly one quarter of that for the other candidates, and that’s very hard to pass up. Granted, I’d be missing out on some features that come with the higher-end solutions, like redundant controllers or volume snapshotting, but this would be acceptable for our purposes (those looking for a complete failover solution might be interested in the Vmirror).
The final point to consider now is whether the Xserve RAID can perform with the reliability of the boxes costing four times more. Even I, an Apple defender to the end, have had my doubts, but when one enterprise software vendor gave me an unsolicited recommendation for Apple storage, I began to consider the Xserve RAID seriously. Most reports I’ve found on the web generally agree that the Xserve RAID, especially for the price, is a solid solution, so I’m becoming convinced. We might very well see a couple “Xserve SANs” in our server room very soon.
I totally disagree with this article, XSan is intended to have concurrent access not only to the same file/folder but even to the same volume on a Xserve Raid system.
I write this because I fully tested both environments, with and without XSan.
We have two Xserve Raid systems and 14 Xserve G5 and we initially thought the same thing: we buy a 12-ports fibre channel switch to attach the two Xserve Raid systems (4 ports) and eight servers (8 ports) so that the eight FC-enabled Xserves will be able to simultaneously access the raid volumes.
We were really disappointed when we discovered that yes, all the raid volumes were mounted on all the Xserves attached to the FC switch, but only one (randomly chosen) Xserve was really and consistently writing its data to the raid volume, all the other servers only got a "fake" access to the raid volume because all data they wrote there vanished into nothing, they neither saw the new/modified files saved by other servers.
I know, this will be a great disappointment for may enthusiast that was thinking to buy some Xserve Raid systems, but unless you have XSan or another similar network file system you can forget about concurrent access to raid volumes.
Chris, you might want to take a look at Sun's Thumper which offers lots of RAIDed disk space in a nice 4U box. If you grab the latest OpenSolaris releases, I hear it will ever run as an iSCSI target. Paul Murphy also demonstrates that you get a pretty good bang for your buck.
This article is misleading in a few was:
1. While you can have one Xserve Raid attached to a FC switch and multiple Xserves mount separate luns on that RAID, you cannot have them just unmount one and mount the next. The the Xserves will just try to mount them all. Since every Xserve tries to mount all the possible LUNS you will end up with file corruption. What you end up having to do is set up LUN masking. This maps a LUN to a particular Xserve. The end result is that a single LUN can only be mounted by a single Xserve. To get the same LUN to mount on another server you would need to manually go back and adjust the LUN masking every time.
2. XSAN does more than allow an Xserve RAID to be mounted concurrently by two servers. XSAN allows multiple RAIDs to be pooled into a single larger storage volume. Even this is an over simplification of XSAN as it is a complete file system.
3. "Xsan system, which requires two Xserve boxes" XSAN does not require 2 Xserves. In fact, it doesn't require any. It does require a system to be a meta data controller and if you are smart you will have a dedicated back up MDC. But these could be Mac Pros, the do not have to be Xserves.
Fab, sorry if it wasn't more clear, but I'm saying the exact same thing as you in the blog: only with Xsan can you concurrently access the same volumes from multiple servers.
Aggregated storage is the big bonus of Xsan. Just add RAIDs, repeat as necessary. The RAID5 Xserve RAIDs are great products, and Xsan is wonderful app. It sells more storage. You always need more.
Sorry Chris, I must have misunderstood the core of your article, but when this morning I read that "you can also plug one [Xserve Raid] into a Fibre Channel switch, and its volumes will be available as local storage to the servers on that switch" my past frustration has come back to my mind.
BTW, we are now (almost) happy XSan users. I write "almost" because we have found that even XSan doesn't fully meet our needs for a stable and fault-tolerant storage solution, and I'm not talking about raid data mirroring, volume snapshots or other advanced topics: I just wish that in the rare occasion of a long electric blackout (so long that our UPS went off) the XSan storage should go automatically up when the power returns instead of staying there to look for some "lost" LUNs, clients and metadata controllers.
Chris,
Actually you can do this and it works fine (but for only one LUN, so this won't work if you need to expand the storage, etc -- on the other hand, its great if you just need a way for your cluster to concurrently crunch on the same data for long streches of time, for instance.) However you have to be sure that no more than one server will mount the volume read-write at a time. If more than one has write access, then, as Sirenoidea points out, they will try to concurrently write to it in an uncontrolled fashion, and that will be very bad news. For our cluster, the fstabs are set up so that there is one priveleged master node with r/w access, and the rest are always read only.
The other problem is that the servers won't see changes to the volume in real-time (I think that if they mount the RAID read-only, they don't expect the contents to change now and then -- I think they think it's just a giant CD-ROM or something!) The only way to make them aware of new or moved files on the RAID is to reboot them.
However this is not only not crazy, but in fact I know that they even do this internally at Apple here and there.
A major thing to keep in mind is that Xsan is a *file system*. In addition to the recommended hardware, one must purchase the Xsan "software" for each Server (or Video workstation) which adds ACFS (Apple Clustered File System) to the supported File Systems in /System/Library/Filesystems. Calling anything else a SAN, IMO, misses the point.