Just realized how fragile a virtual machine can be.
Yesterday I logged into a Windows server and found that some programs were missing (python among others). No big deal. I just assumed that someone else had removed them to free up space or something. Then I found a few files I knew I had both edited and deleted. And the files I knew I had touched suddenly had a timestamp dating back to 2008. Luckily this isn’t a critical production machine, but never the less, how could this happen?
When I checked the vCenter log, I found that the virtual hardware on this particular VM was updated just short of 24 hours earlier. The VM also contained a system generated snapshot dated 2008. The Windows event log, however, where having a gap from 2010 till 2012 (current day).
What has happened is this:
• The VM was powered off.
• The virtual hardware was upgraded.
• When the machine was turned back on, the system reverted back to a previous snapshot (without logging it), thus overwriting the current image.
There are several things here that I find “scary”. The first is that the VM went back using an older system state without even logging it.
The second “scary” thing is that the system generated snapshot was wrongly labeled with the year 2008, when it clearly contained data from 2010 (We actually tested this by reverting to this image just to check. We had nothing to loose anyway).
There is only a few things you need to make sure before you begin, otherwise you might get a nasty surprise when all your virtual machines reboot.
Also, if you are going to change the IP-address on the vCenter Server, make sure that all your ESX hosts can communicate with the new address range.
The ESX hosts should be able to survive without a panic when the vCenter Server goes down for maintenance. But this can depend on how you have configured your vCenter Server.
1. Remove the vCenter Server from old domain
Log into the vCenter Server with Remote Desktop (or via a console) and remove it form the old domain. Reboot.
2. Change IP-address on the vCenter Server (optional)
Next step depends if you actually need to change the IP-address. If you don’t, you don’t. If you do, have in mind that you will lose connection once you click “ok” or “apply”. Don’t forget to change the gateway and DNS-server addresses as well.
In my case I had to change the VLAN bindings on the switch port. We keep out two domains on separate VLANs.
3. Add vCenter Server to new domain
Add the vCenter Server to the new domain. No surprise here either. Complete the process with a reboot.
4. Update configuration files on ESX hosts (may depend on step 2)
The ESX hosts will most likely acknowledge the vCenter Server before you get to this step because of fingerprints and other magic. But nonetheless, log into the ESX server with ssh and edit the vpxa.cfg file. All the commands here has to be executed as the root user.
(Before you start; log into each ESX host with the vSphere Client and add a user with rights to log in remote via ssh.)
(Actually this only applies to those who changed the IP-address in step 2)
Change the IP-address listed between the <serverIP> tags in the file, like so:
Given that the new vCenter Server address is 192.168.20.20. Save using Ctrl+X.
Also, check the hosts file for any references to the old address or domain name.
In my case I changed it from
192.168.10.10 vcenter vcenter.domain.com
to the following
192.168.20.20 vcenter vcenter.domain.lan
and save using Ctrl+X. (Yeah, we change from a .com domain to a .lan domain)
If you have changed the vpxa.cfg file you will also have to restart two management agents on the ESX hosts.
Make sure you haven’t configured auto start/stop on the Virtual Machines at this point. If you have, disable it. Restarting services may lead to an unexpected reboot of all the virtual machines running on the ESX host.
#service mgmt-vmware restart
#service vmware-vpxa restart
5. Restart VMware Virtual Center Service
Log into the vCenter Server and start the services console. Just type “services.msc” in the run dialog.
Find and select the VMware VirtualCenter Server service and restart it. It may also drag another service with itself in the process. No problem.
6. Log in using vSphere Client
Start the vSphere Client and log in using the new domain name and your administrator credentials.
Hopefully everything is up and running and all the ESX hosts are connected. I would advice you to take a look at the KB-articles in the last section before you actually attempt to follow this guid. They will give you a better understanding about how this works.
Now; vCenter Server still contains some references to the old server and/or domain name. So keep reading.
vCenter Still Contains References to the old Server or Domain Name
This can, and mostly will, happen to you too. And the way I found out was when I tried to log into the vCenter Server using vSphere Client and VMware Update Manager (a plugin) complained about being unable to reach the server, referring to the old domain name.
There was an error connection to VMware vCenter Update Manager. The request failed because the server name could not be resolved.
Turns out that the old name is soft coded into different configuration files and registries.
I had to make changes to the following files and places:
VMware vSphere Client
vci-integrity (only if Update Manager is installed)
1. vSphere Client changes
In the menu, choose Administration and vCenter Server Settings.
If you are using a license server, and the service is installed on the same machine as the vCenter Server, you have to check the settings under Licensing.
And under Advanced Settings you may have to change the following keys:
2. Windows Registry
Using regedit from the run menu, navigate to the following places:
Some of the objects here will have unique names, so they will be named differently on your system. So in the simplest form, go through each and every object under OU=ComponentSpecs and OU=Instances and right click and select Properties.
Scroll through the list and edit every attribute that contains a reference to the old server name.
When all this is done, restart the VMware services or just reboot the vCenter Server. I would recommend a full reboot to make sure the all the changes are active.
Found a nice tip for using VMwares Infrastructure Client to manage VMware Server, a product which is usually managed via web interface. The tip is to simply add the port number of the web interface after the server name or ip address in Infrastructure Client. I.e. “servername:8333”, where 8333 is the default port number for VMware Servers web interface.
The reason for this is most likely that VMware Server is installed as an application on top of an operating system. This underlying operating system might very well use the default SSL port number, 443, for its own web service already.
However; If you’re running VMware Server exclusively, it should be possible to change the port number of the web interface to 443, thus avoiding this issue all together. At least my guess is that this will work.
If you want a free hypervisor, but don’t want the trouble of maintaining an underlying operating system, I would recommend taking a look at VMware ESXi. I’ve written a blog post about it. The only reason, as I can see, to run VMware Server is the hardware support, since the operating system takes care of this. VMware ESXi has limited hardware support by default. But there are ways to get support for SATA controllers, among other things.
Also; You might want to check out Citrix XenServer, which also has been released for free. This is a bare-metal hypervisor, meaning you don’t need an underlying operating system. But I don’t have any experience with XenServer, so I can’t really give any advice or tips yet.
My home server, Threepwood, named after the main character in the popular game series “Monkey Island”, has finally taken the step out of this world and into the virtual space.
For a few years it resided within a Compaq Proliant 400 with whooping 384MB of RAM and 450MHz CPU. Running Ubuntu Linux, this has actually been enough for most of the tasks. It has mainly been used to host a few blogs, my brother and mine, thus running Apache2, MySQL and PHP5.
The conversion took about 1 hour, but then again I only had a 40GB hard drive, so times may vary depending on hard drive size and network capabilities.
After the conversion I took the time to upgrade the kernel and other packages. Until today my server had 294 days uptime, so one might say it was due for a new kernel. I also gave it some more memory to play with, 1596MB to be exact, and a 2.40GHz CPU.
VMware ESXi is a free Hypervisor from VMware. In simple terms, a host for virtual machines. The difference from VMware Server, which also is a free Hypervisor, is that ESXi runs directly on the hardware whereas VMware Server need to run on top of another operating system.
When I first started thinking about virtualizing my home server, I initially though about VMware Server. Simply because that was the only free alternative I knew about. It was only when I attended a VMware Infrastructure course a few weeks ago I learned about ESXi. What intrigued me about ESXi was the fact it could boot from a USB pen drive.
But where VMware Servers can use of all the hardware available to the underlying operating system, ESXi has very limited hardware support. However, rules are meant to be broken. Right?
Making a bootable ESXi USB pen drive:
Let’s make a bootable ESXi USB pen drive. The easiest way to do this is using Linux and some more or less standard programs. You can also do this on Windows, just follow this guide from Yellow Bricks which is the guide I used to make this one.
Download the free ESXi ISO-image from VMware. You need to register so you can get your free licence code.
Mount the ISO-file. ( “mount -o loop ESXi.iso /mnt/iso” )
Go to the directory where you unpacked the files, and then some. ( “cd /root/esxi/usr/lib/vmware/installer” )
Unzip the big-file. (” bunzip2 VMware-VMvisor-big-3.5.0_Update_3-123629.i386.dd.bz2“)
Place this image onto your USB stick. This WILL destroy all data on the USB pen drive. Also, you need to know which device your pen drive is. Mine was “/dev/sdb” ( “dd if=VMware-VMvisor-big-3.5.0_Update_3-123629.i386.dd of=/dev/sdb” )
Step 6 will take a while since we’re writing 750MB of data to the USB-stick. When it’s finished you will have stock ESXi installation on a bootable USB pen drive. Congrats!
More hardware support.
As mentioned earlier, ESXi has very limited hardware support out of the box. However, the VMkernel itself has better hardware support which VMware doesn’t wants us to know about. You only need to find out how to enable it.
For my own part I added a 3Com network adapter, since there is no way my onboard Realtek is supported. 3Com and Intel cards should be a safe bet, but don’t take my word for it. Also I wanted to use the computers local SATA disks as datastores. Thus the hacking begin.