February 2, 2009 - Sven Huisman

ESX (whitebox) local storage perfomance tip

I’ve been playing around with my ESX whitebox lately and I had a problem with the write performance of the local storage. Fortunately, I found a solution that I wanted to share. It can be useful for your ESX whitebox or even your production server.

First of all, here are the specs of my whitebox:

Mainboard: Abit IP35 PRO
Processor: Intel Core 2 Duo E8400
NIC: Intel PRO/1000 G T Desktop Adapter
RAM: 2x Kingston KVR800D2N5K2/4G,4096MB, DDR2, PC6400 (total 8GB)
RAID controller: HP P400 with 256 cache
HD’s: 4x 500GB SATA

The first problem I encountered was the mainboard I initially bought (GigaByte GA-EP45-DS3) didn’t recognize the RAID controller. It had 2 PCI-express slots but it appeared that both were meant for VGA. So first check on the internet if you decide to use a server RAID-controller on a desktop mainboard. The card you have in mind may not work with your mainboard.

The second problem I had was having a bad RAM module. I’ve blogged about that issue before.

The third problem was the fact that my mainboard did not recognize my processor correctly. I picked a duocore because I figured I would run 8-10 VM’s max. That’s a 4-5 to 1 CPU-core. It turned out I have a E0 stepping which my mainboard apparently did not support. Therefor I could not enable VT. Fortunately, I found a beta bios update for my mainboard. After flashing the mainboard (it was hard to find a floppy drive and floppy, I can tell you that), everything was ready to run ESXi 3.5 U3 from an USB-key.

Everything looked OK. I installed vCenter server as a VM, couple of domain controllers, fileserver, terminal server gateway (to remote control my VM’s securely over HTTPS) and a couple more. I noticed the write speed on the VMFS volume hosted by the P400 RAID controller would not go beyond 9 MB/sec (red circle). That’s odd, I thought. I have 256 MB cache, so why is it not using it? I also have a single SATA disk connected to the onboard ICH9 controller. This single disk is getting write performance of 50 MB/sec (blue circle).

I did some research and it turns out the HP P400 RAID controller disables the cache for write when there is no battery backup unit connected to the cache and there is no way to turn it on except connecting a battery to the cache. Apparently there are some more vendors that disables the write cache when there is no battery present. It’s pretty logical, because when you have a hardware failure during a write on disk, data could get corrupted. But for us techies, it would be nice to enable write cache on our test-environment.
I ended up buying a battery backed write cache enabler for my RAID controller and I don’t regret it: write performance of 150 MB/sec!

 

 

In conclusion:

If you want decent write performance on your ESX-whitebox, you should:
A) Buy a RAID-controller with battery backed write cache.
B) Buy a RAID-controller on which you can enable write back cache without the need of a battery.

In both cases, do a search on the web if the RAID controller will work on your mainboard.

I’m curious what write performance you get on your ESX whitebox and with what setup.

Virtual Infrastructure ESX / ESX 3.5 / esxi / HP P400 / performance / RAID / VMware / Whitebox /

Comments

  • Frank Korpershoek says:

    Hi Sven,

    You can also have a look with the Array Configuration Utilitiy (not via the bios setup) at other settings for your controller. Default this controller also disables Physical Drive Write Cache and sets the read/write cache ratio to 25/75% With the ACU you can change these settings too.

    regards -frank-

    • Sven Huisman says:

      Well, that was one of my challenges. As I use ESXi and use the RAID-controller in a non-HP server, I cannot use the Array Configuration Utility. I could install windows 2003 on a seperate disk and install the Configuration utility, but I’m affraid to destroy my VMFS-volume. If you know of another way to configure the RAID-controller, I would like to hear it.

      • Frank Korpershoek says:

        I think you mean the acu on a bootable hp smartstart cd? There might be another way too. On a hp system you can also install the array config utility on the service console itself, but I’m not sure if you’ll get this working on a white box system. I also see that you’re running ESXi which might be another hurdle 😉

        regards -frank-

  • mervincm says:

    I have a Similar Whitebox ESX-i System

    Mainboard: EVGA 780i SLI
    Processor: Intel Core 2 Quad Q9550
    NIC: 2 integrated Nvidia plus a dual Intel PRO/1000
    RAM: 4×2048 Mushkin PC6400 (total 8GB)
    RAID controller: HP P400 with 512 cache with Battery
    HD’s: 4x 500GB SATA Seagate ES.2
    BootDrive: USB-Key

    Have you found a way to get HDD LED lights working? in an HP server, you will get one for power, access, and failure. Right now I have none.

    Have you found a way to get raid status or warnings? I am worried because now I have no way to find out about a failed HDD unless I boot up and check in the BIOS.

    Are you running RAID 10 or RAID 5? My guest File server performance is terrible right now. 10-20 MBPS with a minimum amount of activity

    PS to configure the raid controller, I added another HDD, installed windows 2003 server on it, the P400 drivers, then the array config tools from HP, that way I was able to change the read/write cache ratio. I had to use the command line tool, the gui version would not find the controller. Also you cannot use the bootable CD from HP, it refuses. When I get a chance, I am going to try a PE based USBKey, I think the array config tool will work from there.

    PS Gigabyte booting sucks. The P400 will not boot in my X58 either. Also, I was unable to get the USB ESXI booting working on 2 other GA-P35-DSL boards that I own. I didn’t try the P400 in them yet.

    We had a meeting yesterday with our Vmware rep, hoping to get my hands on 4.0 Beta and try some things!

    • Sven Huisman says:

      No, I did not have found a way to get HDD LED lights working.

      No, I haven’t found a way yet to get raid status or warnings.

      I’m running RAID 5 with 4x 500GB SATA disks.

      ESXi 4 works perfectly on the system.

  • john says:

    were you able to get the P400 controler running on your gigabyte X58?(i have one and the system won’t recognize it…)

  • bjorn says:

    it is possible to get the status of the P400 controller on a ESX (not esxi) white box with HP raidcontroller

    just install

    ( rpm -ivh )

    cpqacuxe-8.10-1.noarch.rpm
    hpacucli-8.10-2.noarch.rpm
    hpsmh-2.1.8-177.linux.i386.rpm

    and start the acu web interface

    cpqacuxe -R

    now open the esx firewall ports

    esxcfg-allowincoming
    esxcfg-allowoutgoing

    and browse to the ip of the esx box on port

    2301

    Here you can see the status of the drives and change the array settings …

    I am using hpacucli, cli interface with a script emailing me the array status with sendemail.

  • mervincm says:

    a few updates

    system has been working smoothly. I think my poor fileserver performance might have been while the battery was still uncharged. I have been getting about 40-50 MB/sec read and write to the above 4 drives in RAID5 all the way to my workstation. I recently added 4 seagate 7200.12 1TB drives and in my first test I was able to see double that (!!) on a 4TB ISO

    I still have to look at enableing SATA on drive cache, but as I no longer have my drive with 2003 server and the CLI installed, I would have to redo that.

    BIOS 6.86 and 7.08 are no good on my system!!!
    If I use them, then my array card hangs during boot, just after all the drives are initialized. I had to downgrade back to 5.26 and that meant doing so in a real HP server and the firmware cd on USB key

    ESXI 4 update 1 are working wonderfully off my USB stick

    I was able to boot on my gigabyte x58 with the p400 , they fixed it with a bios update. I can’t recall exactly what version fixed it but thats all it took.

  • Michael says:

    esxi hp array health monitoring for whitebox machines

    http://forum.lettronics.com/forums/thread/1273.aspx

  • Freakin says:

    Thanks for the advice in this post. Got my whitebox running with a P400 on a IP35-Pro

    IP35-Pro
    Q6600
    8GB DDR2 800
    system on 300gb velociraptor
    P400 v5.26 w/ 8x 15kRPM 72GB SAS drives in RAID 1+0. 256MB BBWC @ 75/25

    I feel like I should be getting better read rates off this thing. I’m getting around 130-150MB/s for sequential read and 280MB/s for sequential write. This is on a guest using CrystalDiskMark

    Any way to get better performance out of these 8x 15k SAS drives?

    Also, a sort of weird problem I am having is with the controller not reporting proper port numbers or info for disks. In ACUCLI if I look at physicaldrives it shows a single drive reported as port 0:0 and repeats it 8 times. Same serial, status, everything.

    Once I got the ACU on the SMH running (thanks Bjorn!) I could see them all identified by SCSI ID (but still not port).

    This is on ESX 4.1