Thursday, November 14, 2013

vCenter Server Service stopped unexpectedly: Failure while trying to connect to SSO Admin server



This morning our VMware VirtualCenter Server Service stopped unexpectedly.  My colleague restarted the vCenter server, confirmed that all related services had successfully restarted and was then able to log back into vCenter.  (Thanks Mark!)

I took a look at the vpxd logs and noticed there were several entries regarding SSO.  

 (For Windows Server 2008, the vpxd logs are located in the following directory: C:\ProgramData\VMware\VMware VirtualCenter\Logs)

2013-11-14T10:25:45.273-07:00 [00496 error 'HttpConnectionPool-000001'] [ConnectComplete] Connect failed to <cs p:000000000b633090, TCP:vcentersvr.mydomain.com:7444>; cnx: (null), error: class Vmacore::SystemException(No connection could be made because the target machine actively refused it. )

2013-11-14T10:26:21.446-07:00 [01076 error '[SSO][SsoCertificateManagerImpl]'] [CreateAdminSsoServiceContent] Failure while trying to connect to SSO Admin server: No connection could be made because the target machine actively refused it.  . Will retry in 10 seconds.

2013-11-14T10:27:21.673-07:00 [01076 warning '[SSO][SsoCertificateManagerImpl]'] [CreateAdminSsoServiceContent] Max connection attempts (10) reached. Giving up ...

2013-11-14T10:27:21.673-07:00 [01076 error 'Default'] Failed to intialize VMware VirtualCenter. Shutting down...

This issue is caused by a realtime virus scan putting a lock on files used by SSO.

To resolve this issue, exclude the scanning of the SSO folder:
X:\Program Files\VMware\Infrastructure\SSOServer

Tuesday, November 12, 2013

vSphere ESXi - Low CPU usage and High CPU ready


Recently, I’ve been receiving more requests for VMs with multiple vCPUs. 

A quick look in vCenter shows that the CPU utilization of the ESXi hosts in the cluster as low.  However, you need to dig a bit deeper and take a good look at your CPU ready performance stats to get the entire picture. 

**Low ESXi CPU utilization DOES NOT necessarily mean your CPU ready times are acceptable**

CPU ready is the time that a VM is “ready” to run, but could not be scheduled because of lack of physical CPU resources.  This typically occurs when the physical CPUs are over subscribed on an ESXi host. 

The Low CPU usage and High CPU ready condition can be caused by one or more “Monster VMs” configured with more vCPUs than the ESXi host can handle.  This in turn creates a bottle neck for the remaining VMs on the host.

vCenter gives you the Ready information in Milliseconds.   

Keep in mind that the %RDY number is the sum of all vCPU %RDY(s) for a given VM.  So, 10%RDY for a VM configured with 1 vCPU would mean there is serious contention for CPU resources.  However, if the VM was configured with 4 vCPUs it would mean that 2.5 percent of the time is used waiting to be co-scheduled. In the case of a 4 vCPU virtual machine, the max possible %RDY is 400%.

Conversions (approximate):
1% (%RDY) = 200ms
5% (%RDY) = 1000ms
10% (%RDY) = 2000ms

%RDY rough guidelines:
<2.5%                    OK
5%                          Contention
10%                        Serious Contention

VMWares performance trouble shooting guide has the CPU ready threshold at <1000ms per vCPU.

vCPUs Allocated
CPU Ready Threshold for VM
1
1000ms
2
2000ms
4
4000ms
8
8000ms
 

**Update June 2021** - Recently, I've updated my method to determine if my CPU Ready figures are within an acceptable range.  Below is a fantastic calculator to easily determine if your VM is suffering from contention:

1. Obtain the summation value from vCenter.  No need to ssh into the box. I always used the max value.

2. Use VMCalc.com to determine if you are within the acceptable range. 

3. In this example, I used the data for the “Last Day”, entered the Max summation value and number of vCPUs. 


To resolve this issue:
1. Confirm that all your VMs are "Right Sized".  Use vCenter/vCenter Operations/Turbonomics to see if any VMs are over-provisioned.  Reduce vCPUs as needed.
2. Enable DRS on the cluster to have VMware migrate the VMs to the appropriate host.
3. For Dell R810s confirm that power management is set for "Maximum Performance".  For R820s, set system Profile to "Performance".  (this killed us on an SQL cluster).
4.  If your cluster is overcommited, you may need to upgrade or add an additional host.

Monday, October 21, 2013

PowerCLI: How to Enable SSH on an ESXi host

I recently had to SSH into an ESXi host to perform some troubleshooting tasks.  SSH can be enabled  on a host through the Direct Console User Interface (DCUI) or the vSphere Client.  However, a much more efficient method is to leverage PowerCLI:

How to Enable SSH on an ESXi host:
Get-VMHost myesxihost.mycompany.com | Get-VMHostService | Where {$_.Key -eq "TSM-SSH"} | Start-VMHostService
After performing what I needed to do, I proceeded to disable the service with the following:

How to Disable SSH on an ESXi host:
Get-VMHost myesxihost.mycompany.com | Get-VMHostService | Where {$_.Key -eq "TSM-SSH"} | Stop-VMHostService

 Waaaay faster!

Tuesday, August 20, 2013

Event ID: 4292 The IPSec driver has entered Block mode. - Server initially pingable during startup, then stops

This morning I had a VM which was inaccessible over the network.  The VM was "up" and I was able to log into the server locally using the console.  IP settings were good, I confirmed that the network settings on the VMware side were good, and I did the ol' disconnect/reconnect of the vNIC.  No Go.

One interesting thing,  upon reboot, I would receive ping replies, then they would abruptly stop:

The Event logs had the following:

Event ID: 4292 The IPSec driver has entered Block mode


Apparently, the IPsec service on the server 'lost its mind" and placed the NIC in a blocked state.  The resolution according to Microsoft KB 912023 is as follows:

To resolve this issue, follow these steps:
  1. Delete the local policy registry subkey. To do this, follow these steps:
    1. Click Start, click Run, type regedit in the Open box, and then click OK.
    2. In Registry Editor, locate and then click the following subkey:
      HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows\IPSec\Policy\Local
    3. On the Edit menu, click Delete.
    4. Click Yes to confirm that you want to delete the subkey.
    5. Quit Registry Editor
  2. Rebuild a new local policy store. To do this, Click Start, click Run, type regsvr32 polstore.dll in the Open box, and then click OK.
  3. Verify that the IPSEC Services component is set to automatic, and then restart the domain controller.
In my case, the registry did not have the \IPSec\ Key...  So, I ended up just running following command and restarting the server:

regsvr32 polstore.dll

I bounced the server several times to confirm that the server came up clean and the issue had been resolved.

Friday, August 16, 2013

PowerCLI: How to Find VMs with Change Block Tracking (CBT) Enabled

Our previous backup solution had a  interface which conveniently displayed which VMs had Change Block Tracking (CBT) enabled.  Unfortunately, our new solution does not...

To quickly find the VMs which have CBT enabled, run the following "one-liner" against your vCenter server:

Get-VM | Get-View | Sort Name | Select Name, @{N="ChangeTrackingStatus";E={$_.Config.ChangeTrackingEnabled}} > c:\cbt.txt


Depending on how large your environment is, it may take several minutes to complete.  I piped the output to a text file for documentation purposes.   Here's an example of its output:


Thursday, August 15, 2013

How to free up disk space in Windows 7 and Windows Server 2008 R2

I recently needed to free up a little space on the OS partition of a Windows Server 2008 R2 SP1 server.  One way to reclaim space is to remove the Service Pack files after a successful SP installation.

**Warning** Confirm that the Service Pack install was successful.  The following command will make the service pack permanent.  An uninstall will no longer be possible.

Run the following command against an online installation:

dism /online /cleanup-image /spsuperseded

 
This process takes several minutes to complete, but freed up close to 4GB of disk space.  This should hold me over until the server is ready to be decommissioned.