April 12, 2011 - Matthijs Haverink
Disappearing Virtual NIC & VMX has left the building
I know I’m not the first who noticed it, since this message is to be found in log files for years with ESX, but I still wanted to share this one:
Especially when you’re seriously troubleshooting an issue and this message appears, it’s a good laugh. Kudos to the VMware programmer with humour :).
Back 2 Business: the missing/disappeared Virtual NIC
This was a new one for me so I’d like to share this one.
What happened was that a user called about his VM being unreachable. I also established that and used the console to view the IP config: it was dead empty; no adapter; nothing.
I was ready to blame one of my colleagues for being thick by removing a nic from a live production VM but then established through vCenter logs that it wasn’t done using the vSphere console by a VI admin.
I then took a look at the vmware-**.log files on the datastore for that VM and discovered this message:
vmx| Powering off Ethernet0
vmx| Hot removal done.
Okay so this was a hot removal but not by a VM admin; how is that ?
And there you’ll find the explanation of the amazing HotPlug feature introduced in ESX(i) 4…
So the cause of the disappeared NIC was that a user with administrative privileges within a Windows VM gets this dandy handy HotPlug feature from VMware and the user was able to delete the NIC in just 2 simple clicks:
This gives any administrator within the guest OS the ability to fully remove the NIC from the VM config! Why is this feature is turned on by default ?! This is, in my opinion, a great risk in a virtual environment. Sure you’re right when you say; you should think twice before giving people full admin privileges on the OS. But in the real world, there are always people with administrative privileges that have little knowledge of what mayhem they can cause. This HotPlug feature being enabled by default doesn’t really help the situation.
But just disable it and you’re done you might think; this is partially true. If you have one VM that’s doable. But the procedure is that you have to power off the VM and then fix the VMX file by adding devices.hotplug = “false” to it. This really does not make me happy for an environment with hundreds of VM’s. There is a suggested registry hack that might do the trick but even that seems to be too tricky to push with a GPO.
I really wonder why this feature is enabled by default anyways: you don’t let a user of a server in your datacentre to remove a physical NIC from a physical server, right ? Why then give them that possibility in a virtual environment? In my opinion the HotPlug feature can do more harm than good. If you do have real use cases for this feature; please let me know; I love to learn!
Nevertheless, I’m afraid this is going to cost me quite some effort to disable this feature for all current VM’s in the datacentre. If you have any suggestions on how to fix this in a scripted though controlled way I’m all ears!