February 2, 2009 - Sven Huisman
ESX (whitebox) local storage perfomance tip
I’ve been playing around with my ESX whitebox lately and I had a problem with the write performance of the local storage. Fortunately, I found a solution that I wanted to share. It can be useful for your ESX whitebox or even your production server.
First of all, here are the specs of my whitebox:
Mainboard: Abit IP35 PRO
Processor: Intel Core 2 Duo E8400
NIC: Intel PRO/1000 G T Desktop Adapter
RAM: 2x Kingston KVR800D2N5K2/4G,4096MB, DDR2, PC6400 (total 8GB)
RAID controller: HP P400 with 256 cache
HD’s: 4x 500GB SATA
The first problem I encountered was the mainboard I initially bought (GigaByte GA-EP45-DS3) didn’t recognize the RAID controller. It had 2 PCI-express slots but it appeared that both were meant for VGA. So first check on the internet if you decide to use a server RAID-controller on a desktop mainboard. The card you have in mind may not work with your mainboard.
The second problem I had was having a bad RAM module. I’ve blogged about that issue before.
The third problem was the fact that my mainboard did not recognize my processor correctly. I picked a duocore because I figured I would run 8-10 VM’s max. That’s a 4-5 to 1 CPU-core. It turned out I have a E0 stepping which my mainboard apparently did not support. Therefor I could not enable VT. Fortunately, I found a beta bios update for my mainboard. After flashing the mainboard (it was hard to find a floppy drive and floppy, I can tell you that), everything was ready to run ESXi 3.5 U3 from an USB-key.
Everything looked OK. I installed vCenter server as a VM, couple of domain controllers, fileserver, terminal server gateway (to remote control my VM’s securely over HTTPS) and a couple more. I noticed the write speed on the VMFS volume hosted by the P400 RAID controller would not go beyond 9 MB/sec (red circle). That’s odd, I thought. I have 256 MB cache, so why is it not using it? I also have a single SATA disk connected to the onboard ICH9 controller. This single disk is getting write performance of 50 MB/sec (blue circle).
I did some research and it turns out the HP P400 RAID controller disables the cache for write when there is no battery backup unit connected to the cache and there is no way to turn it on except connecting a battery to the cache. Apparently there are some more vendors that disables the write cache when there is no battery present. It’s pretty logical, because when you have a hardware failure during a write on disk, data could get corrupted. But for us techies, it would be nice to enable write cache on our test-environment.
I ended up buying a battery backed write cache enabler for my RAID controller and I don’t regret it: write performance of 150 MB/sec!
If you want decent write performance on your ESX-whitebox, you should:
A) Buy a RAID-controller with battery backed write cache.
B) Buy a RAID-controller on which you can enable write back cache without the need of a battery.
In both cases, do a search on the web if the RAID controller will work on your mainboard.
I’m curious what write performance you get on your ESX whitebox and with what setup.