Recently, we were working to bring up a VMware installation for a client of Puryear IT and we hit a snag. To provide some background first though: We had decided to go with a GigE NAS based environment rather than a more traditional SAN. We had seen Dell’s NF500 in use already and were pleased with it overall, so we went with the NF500 with RAID-10 on a GigE switch and, of course, GigE on the Dell servers running VMware.
Great, right?
Alas, not so much. During our benchmarking, we found that the NFS performance on the NF500 across the GigE was pretty bad. This goes for every variation, including NFS over UDP and TCP, v2 and v3, rsize and wsize of everything from 4kb to 32kb, and so forth. Yes, we tried every performance tweak in the book, but just could not get the Linux servers to get good NFS performance against the NF500. Well, the performance is good enough if you were using the NAS as only a file server, but not if you want to run VMs off it.
That said, there is no real reason why you can’t or shouldn’t run VMs off a GigE network and a really fast NAS. It’s more than sufficient. Unfortunately, we didn’t have the hours to troubleshoot whether there was something going on with the Linux NFS implementation or the NF500, but we still had to get the problem solved.
Fortunately, we did find a solution that not only worked, but worked well: CIFS.
Everybody knows and loves CIFS. (Well, everybody at least knows CIFS. Oh, and hello Samba.) It’s just Windows Networking. Generally, we use NFS within Linux and UNIX networks where we can tighten down security enough on the network to make it reasonably safe to use (NFS is not, and has never been, a secure protocol.) But I am quite familiar with CIFS and was curious if using it would clear the problem up. And yes it did.
I found that mounting the VM shares off the NAS on the local Linux VMware servers let us transfer at near-wire speed. We were then able to run our VMs off the NAS; we have yet to see any performance issue or bug, and the whole thing just works like a champ.
Very interesting.
From Dell's web site: "The PowerVault NF500 comes pre-installed with Microsoft Windows Storage Server 2003 R2"
Of course it couldn't possibly be Microsoft poorly implementing their NFS support. Nah, must be something else.
Actually, I never said it was NOT Windows, or the NF500, or Linux. I simply said that NFS wasn't performing well enough to work. ;)
> NFS is not, and has never been, a secure protocol.
FUD FUD FUD FUD.
Check NFSv4.
TCP port 25 (SMTP) is the source and destination of all spam. and the port25 blog is the source and destination of all FUD.
NFSv4 does resolve many of the security issues inherent in NFSv2 and NFSv3, but.. NFSv4 isn't exactly fully supported or deployed.
Not that I had meant to harp on this issue, but I have never followed the line of thought that something that isn't really fully out there makes something true or untrue.
NFS has a history of being insecure. That's a fact. So can we find other network filesystems out there to replace it? Yes. CIFS provides some security features that NFSv3 doesn't (bring up NFSv4 when it's OUT THERE and fully supported) but it doesn't really work on UNIX for everything because of the CIFS ACL issue. What about Andrew? Or others?
The "FUD" rant is, frankly, pretty pointless, especially in light of the fact that NFS does indeed have problems. ;)
Uhh.... CIFS has a history of being insecure too.
For example, early server implementations (MS) didn't mandate password validation. So if a client connected, and didn't login, it could access everything. It was bug compatible with MS clients, because they would always login.
There is the password hash issue. NFS has the advantage of being a server-to-server protocol, so no user passwords.
What about all of the DoS issues against port 137 and port 139? Remember the Land attack? When was it possible to kill a Unix server by crafting a corrupted NFS request? I've never heard of such a thing.
But the biggest issue I have with CIFS, is how useless it is for server-to-server file sharing. Everything try to setup IIS to serve files from a CIFS share? Every share requires a separate CIFS session! So if you have 1,000 sites, you need a 1,000 CIFS sessions. Not scalable at all. NFS has no issue with this.
But generally, NFS performance on Windows Server is not great. It is obvious, since MS never optimized this code. The CIFS server code has been optimized for years, and doesn't it basically run all in-kernel now?
What puzzles me about this, is why aren't you using iSCSI instead? Why use a file server to export file shares contained a big disk image file? Whey not mount the disk and use an actual block-access protocol for it instead? Lot less overhead. The latency of CIFS is going to be a killer.
Hey Tom, thanks for your thoughtful response.
>> Uhh.... CIFS has a history of being insecure too. For example, early server implementations (MS) didn't mandate password validation. So if a client connected, and didn't login, it could access everything. It was bug compatible with MS clients, because they would always login.
I’m going to have to say FUD on this one. Bringing up an insecurity this old just doesn’t make sense. That’s like saying that NFSv1 had problem XYZ and so shouldn’t be used. We aren’t talking about out-of-date versions, but versions in use now.
Which brings us to this: NFSv3 is what is used now. NFSv4 is available, yes, but not widely deployed. I just don’t think using NFSv4 stands up in this argument because it’s not widely in use or even fully supported on Linux. You can even read that here:
http://www.citi.umich.edu/projects/nfsv4/linux/
>> There is the password hash issue. NFS has the advantage of being a server-to-server protocol, so no user passwords.
Wow. I have never seen this argued as a pro for NFS. One of the inherent weaknesses of NFS is the lack of built-in authentication. There are RFCs that even talk about this.
>> What about all of the DoS issues against port 137 and port 139? Remember the Land attack? When was it possible to kill a Unix server by crafting a corrupted NFS request? I've never heard of such a thing.
You can DoS just about anything, including NFS.
>> But the biggest issue I have with CIFS, is how useless it is for server-to-server file sharing. Everything try to setup IIS to serve files from a CIFS share? Every share requires a separate CIFS session! So if you have 1,000 sites, you need a 1,000 CIFS sessions. Not scalable at all. NFS has no issue with this.
Now that is a valid point.
>> But generally, NFS performance on Windows Server is not great. It is obvious, since MS never optimized this code. The CIFS server code has been optimized for years, and doesn't it basically run all in-kernel now?
nfsd runs mostly in kernel on Linux as well. I’m not getting your point.
>> What puzzles me about this, is why aren't you using iSCSI instead? Why use a file server to export file shares contained a big disk image file? Whey not mount the disk and use an actual block-access protocol for it instead? Lot less overhead. The latency of CIFS is going to be a killer.
Now that’s the fun part!
There is a big move toward NFS and network file systems in general for these types of things. iSCSI is great, but the whole SAN mentality is cumbersome. Even some of the guys at Network Appliance are moving toward a network file system mentality:
http://storagefoo.blogspot.com/2007/09/vmware-over-nfs.html
Interesting stuff.
So you used NFS and Linux and had problems. Hmm, you don't say.... ;)
You're really going to have to dig into the TCP/UDP layer to see if there's any latency when dealing with high I/O like that.
Otherwise, I'd also recommend switching to Solaris and using their NFS stack, which has stronger NFSv4 support, among other things.
NFS and Netapp anyone? I run very large Oracle DB's with RHEL 2.6 NFS and netapp and have not any problems. Actually faster then FC to a DELL|EMC Clarrion.
Yes, NFS over NetApp. :)
I agree.
I like SANs, but they're just a lot more work.
What are you running your NFS over for the NetApp? GigE or better?
NFS is much faster then any CIFS benchmark over NAS.
I would recommend trying a clustered NAS (www.exanet.com).
Or a 10G once its available.
Please remember to have enough spindles.