Archive for July, 2012

How to hide warnings when there is no HA management network redundancy.

July 26, 2012 Leave a comment

When HA is enabled in a cluster, if there is no management network redundancy, the vSphere client shows an alarm icon on the host in the inventory, and displays a warning on the hosts summary page.  In this blog we look at how to prevent these from being displayed.

First of all let’s see the effect of turning on HA in our cluster, and with only one management network you can clearly see the warning message.

To disable this message from being displayed, edit the cluster settings, click on vSphere HA, then select the Advanced Settings button, click at the top to edit a row and enter the variable name das.ignoreRedundantNetWarning and enter the value true.  If you then turn off HA and back on again, the warnings will not display.

It should be noted that these warnings are there for a reason, it is best practice to have redundant management networks to ensure redundancy for heartbeat traffic.

How to hide warnings shown when the esxi shell and ssh are enabled.

July 26, 2012 Leave a comment

When remote access to the ESXi shell is enabled (SSH) or when the local ESXi shell is enabled, the vSphere client shows an alarm icon on the host in the inventory, and displays a warning on the hosts summary page.  In this blog we look at how to prevent these from being displayed.

First of all let’s see the effect of turning on the local ESXi shell and remote access.  In the client you can clearly see the icon on the host and the warnings on the summary page shown.


To disable these from being shown, click on the hosts configuration tab, then select software-advanced settings.

The value that needs to be altered is within the UserVars category, and is at the bottom of the list.  UserVars.SuppressShellWarning should be set to 0 to display the warnings and 1 to display them.


As soon as you have made the change to the variable the display will change, even though the ESXi shell or remote access to it is still enabled.


It should be noted that these warnings are there for a reason, it is bad practice to enable and leave these services on as it represents a major security risk.

Categories: vSphere

How do I reset an ESXi 5.0 unknown root password?

July 24, 2012 1 comment

In previous vSphere versions it was relatively easy to reset the root password for an ESX host by booting the server in single user mode, this gave a root access command prompt from which you could change the password.

In vSphere 5, we only have the ESXi hypervisor which does not have the ability to boot into single user mode, so in this blog we look at how to change the root password for an ESXi host.

Firstly, it is worth noting that the only supported way of resetting a forgotten ESXi root password is to reinstall ESXi!

In ESXi as with ESX and linux systems, there are 3 files that control local user accounts.  Found in /etc, they are passwd, shadow and groups.  The passwd historically stored the user accounts and passwords but was found to have security implications as all users could read the file.  Although users could not see what other users passwords were (as they are encrypted in the file), they were able to change their password and compare the encrypted password with that of other users (and more importantly root), if the passwords were the same, they had guessed correctly!  So as a consequence, passwords were removed from the passwd file, and were placed in a second shadow password file.  This file could not be accessed by normal users and hence was far more secure.  As its name suggests, the groups file contains groups.

So on ESXi, the trick to changing the password of the root user is to manipulate the shadow password file.

In order to reset the root password, you will need a bootable Linux cd (or iso if using a lights-out technology i.e. iLo or Drac), any “Live” (runs direct from CD) version should work (I use SuSE Linux Enterprise 11 and boot using the Rescue System option), plus you will follow the procedure better if you have some basic Linux/Unix experience.

First, boot your ESXi server with a Linux live CD or from a USB stick.

Mount the /dev/sda3 partition to /mnt by using the command:

mount /dev/sda3 /mnt

Unzip the state.tgz file to /tmp, it contains one file called local.tgz with the following commands:

cd /tmp
tar zxvf /mnt/state.tgz

Unzip the local.tgz, and change to the etc folder using the following commands:

tar zxvf local.tgz

cd etc

Using VI edit the file etc/shadow to change the password.

vi etc/shadow

The shadow password file has each user entry per line, and the second parameter (after the 1st being the user name) is the encrypted password.  The easiest thing to do is to delete the string of text between the first and second colon, thus removing a password altogether.

Recreate the zip files, and copy the modified state.tgz back to the original partition.

rm local.tgz
tar czvf local.tgz etc
tar czf state.tgz local.tgz
mv state.tgz /mnt/

Reboot your ESXi host, and you should now be able to log in with no password.

Categories: vSphere

Locked out of vCenter Server?

July 23, 2012 Leave a comment

Using vSphere 4 (and later) vCenter Server it is possible when changing permissions to accidentally reduce permissions for the Administrator, thereby “locking” yourself out.  In this blog we look at how to restore administrative access without the need to reinstall vCenter Server.

For the fix to be possible you need a SQL browser/viewer such as Microsoft SQL Management Studio for the version of SQL that you are using (e.g. Express or full blown).

Having installed SQL Studio, stop the vCenter Server service and any dependent services.

Using SQL Studio connect to the vCenter database using the credentials that vCenter uses to connect (or any credentials valid to administer the database).

Right-Click on dbo.VPX_ACCESS and select Open Table.

You should see a row with:

ID: 1                                                   the first entry

PRINCIPAL: Administrators      a local resource on the server i.e. the local Administrators group

ROLE_ID: -1                                        Administrator permissions

ENTITY_ID: 1                                      permission granted at root / vcenter


either edit the first row as above, or if the permission has been deleted, add a row (the ID should be the next available unused number).

If permissions are totally messed up, delete all entries in the dbo.VPX_ACCESS table, and create the above entry.

Save the change and restart the vCenter Server service, you should now be able to log in.

For further info check out the community posting Reset permissions in VMware vCenter Server.

In addition, if the vCenter database has performance or capacity issues, check out the Investigating the health of a vCenter Server database knowledgebase article.

Categories: vSphere

Enabling vMotion for VMs connected to Internal Only Virtual Switches

July 20, 2012 Leave a comment

In many of the VMware courses it is stated that if a virtual machine is connected to a “Internal Only” virtual switch is cannot be migrated to another host with vMotion migration.  This is because a check is performed on the virtual machine network configuration to ensure that it will be accessible after the move, and when connected internally that would not be the case, and from a production perspective makes perfect sense.  There would be no point in migrating a virtual machine from one host to another if it could not then communicate with other virtual machines also connected to the same internal network.

From a test and development perspective, this may be desirable.  For example, you may be testing a multitier application (using more than one vm), and resources on the current host (due to other factors) become scarce, migration of the multitier application to another host with sufficient resources would be desirable.

This can be achieved by disabling the test performed when a vMotion is initiated that looks for connection to an internal network.  (I would not recommend that this is done with Production environments!)

From a vSphere Client connected to the vCenter Server navigate to Administration > vCenter Server Settings > Advanced Settings and add config.migrate.test.CompatibleNetworks.VMOnVirtualIntranet with a value of “false”.

This change does not require a restart of vCenter Server.

Obviously to change the setting back, change the value to “true”.

For more detail check out the VMware Knowledgebase Article 1003832.

Categories: vSphere

Should I disable the balloon memory driver?

July 17, 2012 Leave a comment

For some time during most of the VMware courses that I teach, the question of whether the balloon memory driver should be disabled has been raised.  I thought that it was something that warranted a blog, so here it is!

Although it is always desirable to have more than sufficient resources than required, we have to accept that the whole purpose of virtualisation is consolidation, and therefore to save money.  Virtualisation is therefore “sold” as a concept to organisations with the premise that over commitment is possible.  Unfortunately, many organisations do take this too far, and attempt to run far too many virtual machines with far too little resources, thus resulting in poor performance.

To allow for over commitment  of memory resources, vSphere has a number of memory “reclamation” techniques (as VMware put it!).

Transparent Page Sharing

This is  a background process running on each ESX/ESXi host that looks to see if any VMs have duplicate pages in RAM, the whole idea is to reduce the overall RAM requirements between VMs by only hold one copy and not multiple copies if they hold the same data.  It’s really a similar concept for RAM as disk deduplication in storage.


When the VMware tools are installed in the guest OS within the VM, one component is the balloon memory driver.  It’s job when instructed by the vmkernel is to “reclaim” the physical RAM that has been allocated to the VM but has not been passed back to the hypervisor.

Memory compression

A new feature in vSphere 4.1, this is a technique when memory is very scarce, the idea is to compress pages in RAM rather than use vmkernel swap files, i.e. use disk instead of RAM.

Host swapping

Each virtual machine has a vmkernel swap file (2 with vSphere 5), which is a failsafe, if all else fails and there simply is no more RAM, use disk instead.

From a performance perspective alone, they are listed in order, with Transparent Page Sharing not really having any effect, yet  reducing overall memory requirements, to Host swapping being the option to avoid.

So now let’s have to look at why memory over commitment is a problem.

On a physical PC, the OS is responsible for controlling memory resources, and will use memory for itself, and will allocate memory to processes.  When these processes finish using the memory, it is either handed back to the OS, which then marks it as free or rather deallocated, or the process or rather application may hang on to the memory for future use.  The important thing is that when memory is released back to the OS it is treated slightly differently to free memory that has never been used, in that when asked for memory the OS will favour this deallocated memory.  This stems back to the good old days when memory was not that reliable, the fact that the memory is deallocated must mean that it has been used successfully by a process previously and therefore it is safe to use again.  Free memory in implication has never been used and therefore it’s integrity cannot be guarenteed.

So, given what happens on a physical PC, what’s the story with a VM?

Well, the hypervisor will do the job of allocating physical RAM when the VM tries to use the memory it is configured with, and will continue to allocate physical RAM based of course on the virtual machine’s settings (reservation and limit or configured amount whichever is the lower).

The problems will start either if there is a limit set on the amount of physical RAM the virtual machine is allowed to use being  lower than the configured amount for the VM, or if the physical host is starting to run out of physical RAM to allocate.  In either event,  the hypervisor may be forced to issue memory from the vmkernel swap file (i.e. disk), if it either is not allowed to provide any more physical RAM (when a limit is set on the VM), or has no more physical RAM to allocate as the host has none free.   Having the hypervisior (vmkernel) swap to disk is a failsafe, a get out of jail free!  The VMs will continue to run if they are using disk instead of RAM, but it’s fairly obvious that their performance will be dire or as VMware put it “Performance will be noticeably slow”!.

So how do I avoid Host swapping / use of the vmkernel swap file?

You have two options, ensure that the host or VM has sufficient physical RAM to meet requirements, or allow the balloon memory driver to claw back memory from VMs so over commitment is possible.

So what’s the issue with the balloon memory driver anyway?

When the balloon memory driver is instructed to reclaim memory from the guest, it starts requesting memory from the guest OS, which in turn is given back to the hypervisor for it to use or pass on to other VMs requesting additional memory.  As a consequence, the guest OS will typically start to use its own memory management processes as it will see less available memory due to the driver, the end result will typically be that the guest OS pages more.  This obviously can affect the performance within the guest.

However, it is extremely important to understand that any performance degradation within the guest and application as a result of the guest paging more will be much lower than Host swapping.  The reason for this is that the guest will choose which memory pages are paged to disk based upon their use, and will target pages that have not been used for some time.  Host swapping does no such thing, the hypervisor is responding to a memory request by the VM, and has no idea of what the memory it then allocates will be used for.

One of the best blogs I have found on the subject is by Frank Dennerman Senior Architect in the Technical Marketing team at VMware, and co-author of VMware vSphere 5 Clustering technical deepdive and vSphere 4.1 HA and DRS technical deepdive.

Above all VMware do not recommend that the balloon memory driver should ever be disabled, and that the VMware Tools should be installed (and up to date) in every VM.

Categories: vSphere

Auto Deploy GUI: Fling

Recently it was my pleasure to teach Massimiliano Daneri (Max) from VMware, who works in the Center of Excellence group.

During the course he showed us his great new tool for managing Auto Deployment of ESXi 5.0 hosts.

You can check out the tool at

Categories: Flings, vSphere