vCenter Server Service stopped unexpectedly: Failure while trying to connect to SSO Admin server

This morning our VMware VirtualCenter Server Service stopped unexpectedly.  My colleague restarted the vCenter server, confirmed that all related services had successfully restarted and was then able to log back into vCenter.  (Thanks Mark!)

I took a look at the vpxd logs and noticed there were several entries regarding SSO.  

 (For Windows Server 2008, the vpxd logs are located in the following directory: C:\ProgramData\VMware\VMware VirtualCenter\Logs)

2013-11-14T10:25:45.273-07:00 [00496 error 'HttpConnectionPool-000001'] [ConnectComplete] Connect failed to <cs p:000000000b633090, TCP:vcentersvr.mydomain.com:7444>; cnx: (null), error: class Vmacore::SystemException(No connection could be made because the target machine actively refused it. )

2013-11-14T10:26:21.446-07:00 [01076 error '[SSO][SsoCertificateManagerImpl]'] [CreateAdminSsoServiceContent] Failure while trying to connect to SSO Admin server: No connection could be made because the target machine actively refused it.  . Will retry in 10 seconds.

2013-11-14T10:27:21.673-07:00 [01076 warning '[SSO][SsoCertificateManagerImpl]'] [CreateAdminSsoServiceContent] Max connection attempts (10) reached. Giving up ...

2013-11-14T10:27:21.673-07:00 [01076 error 'Default'] Failed to intialize VMware VirtualCenter. Shutting down...

This issue is caused by a realtime virus scan putting a lock on files used by SSO.

To resolve this issue, exclude the scanning of the SSO folder:

X:\Program Files\VMware\Infrastructure\SSOServer

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2062622

vSphere ESXi - Low CPU usage and High CPU ready

Recently, I’ve been receiving more requests for VMs with multiple vCPUs. 

A quick look in vCenter shows that the CPU utilization of the ESXi hosts in the cluster as low.  However, you need to dig a bit deeper and take a good look at your CPU ready performance stats to get the entire picture. 

**Low ESXi CPU utilization DOES NOT necessarily mean your CPU ready times are acceptable**

CPU ready is the time that a VM is “ready” to run, but could not be scheduled because of lack of physical CPU resources.  This typically occurs when the physical CPUs are over subscribed on an ESXi host. 

The Low CPU usage and High CPU ready condition can be caused by one or more “Monster VMs” configured with more vCPUs than the ESXi host can handle.  This in turn creates a bottle neck for the remaining VMs on the host.

vCenter gives you the Ready information in Milliseconds.   

Keep in mind that the %RDY number is the sum of all vCPU %RDY(s) for a given VM.  So, 10%RDY for a VM configured with 1 vCPU would mean there is serious contention for CPU resources.  However, if the VM was configured with 4 vCPUs it would mean that 2.5 percent of the time is used waiting to be co-scheduled. In the case of a 4 vCPU virtual machine, the max possible %RDY is 400%.

Conversions (approximate):

1% (%RDY) = 200ms

5% (%RDY) = 1000ms

10% (%RDY) = 2000ms

%RDY rough guidelines:

<2.5%                    OK

5%                          Contention

10%                        Serious Contention

VMWares performance trouble shooting guide has the CPU ready threshold at <1000ms per vCPU.

vCPUs Allocated	CPU Ready Threshold for VM
1	1000ms
2	2000ms
4	4000ms
8	8000ms

 

**Update June 2021** - Recently, I've updated my method to determine if my CPU Ready figures are within an acceptable range.  Below is a fantastic calculator to easily determine if your VM is suffering from contention:

VMCalc.com

1. Obtain the summation value from vCenter.  No need to ssh into the box. I always used the max value.

2. Use VMCalc.com to determine if you are within the acceptable range. 

3. In this example, I used the data for the “Last Day”, entered the Max summation value and number of vCPUs. 

To resolve this issue:

1. Confirm that all your VMs are "Right Sized".  Use vCenter/vCenter Operations/Turbonomics to see if any VMs are over-provisioned.  Reduce vCPUs as needed.

2. Enable DRS on the cluster to have VMware migrate the VMs to the appropriate host.

3. For Dell R810s confirm that power management is set for "Maximum Performance".  For R820s, set system Profile to "Performance".  (this killed us on an SQL cluster).

4.  If your cluster is overcommited, you may need to upgrade or add an additional host.