vSphere 6 - SSL Certificates - Overview and Best Practices

As corporate security becomes a higher priority, I just wanted to give a quick rundown on SSL Certificates in vSphere 6 since it has changed drastically from previous versions.

The VMware Certificate Service is part of the Platform Services Controller (PSC)

Key Terms:

VMWare Certificate Authority (VMCA) – Certificate authority for vSphere components only.  A single point of contact for vSphere Certificate needs.  Issues certificates for VMware solution users, machine certificates for machines on which services are running and ESXi host certificates.  Operates in the PSC.  Certificates are managed by the certificate-manager utility. 

VMware Endpoint Certificate Store (VECS) – Serves as a local repository for certificates, private keys and other certificate information.  Runs in vCenter Server Node. 

Types Of Certificates Used by vSphere

•  Machine Certificates- For Secure connections.  This is what causes the web browser certificate warning if the certificate used is self signed. (ex. vSphere Web Client - vCenter server and external PSC have them)
  
• Solution user certificates - authentication of services to vCenter SSO. (ex vcenter service (vpxd))
•  ESXi certificates – provisioned when the host is added to vCenter. Stored locally on ESXi host.

Certificate Deployment Types:

• VMCA Default – By default, the VMCA uses a self-signed root certificate.  The VMCA is then the CA for all VMware components.
• VMCA Enterprise -  The VMCA is used as a subordinate/Intermediate CA and is issued a subordinate CA signing certificate.  It can now issue certificates that trust up to the enterprise CA’s root certificate.  Not accepted by most security groups, since this poses a security risk. 
• Custom Certificates – The VMCA is bypassed. Need to issue a enterprise/3rd party cert for every component. Must replace each certificate explicitly.  Administrative nightmare! 
• Hybrid - The VMCA supplies some of the certificates, but also uses custom certificates for other parts of the VMware infrastructure.  As of the time of this writing, this is the RECOMMENDED approach.

Hybrid Deployment Details:  

In a vast majority of environments, the following hybrid deployment is the best fit.  

Trusted Certificates are used for the Machine Certificates of the vCenter server and external PSC.  The management interfaces are using a 3rd party/Corporate trusted CA.  These are the most important certificates and is the only user-exposed certificates.

VMCA certificates are the used for the Solution user and ESXi certificates.

Added bonus, no more "Not Secure" warnings in your browser!

VMware KB regarding replacing machine certificates:

https://kb.vmware.com/s/article/2112277

Fantastic VMWare Walkthrough for SSL cert replacement on your vCenter server and External PSC:

https://featurewalkthrough.vmware.com/t/vsphere-6-5/ssl-certificate-replacement-hybrid-mode/

How to mount an ISO to an ESXi Host

I was recently working on and issue in an environment that required me to mount an ISO to an ESXi host.

To access the contents of a CD or ISO within your ESXi Host perform the following:

1. Mount the ISO using iDRAC / iLO etc.
2. Console into the ESXi host and run the following to load iso9660:

vmkload_mod iso9660

3. To find the path to the CD ROM drive run the following:

esxcfg-mpath -l | grep -i cd-rom

4. Run the following to mount the CD-ROM drive:

vsish -e set /vmkModules/iso9660/mount <devicename>

5. You will now be able to see the contents of the CD- Rom in /vmfs/volumes/.  In my case, the ISO I mounted was named "20180215":

6.  Once you're done unmount the CD/ISO with the following:

vsish -e set /vmkModules/iso9660/umount <devicename>

7. To unload the iso9660 module":

vmkload_mod -u iso9660

On a similar note, I've had a lot of success using the following application to create ISOs from files/folders:

http://isorecorder.alexfeinman.com/#/

VMWare Update Manager - interface com.vmware.vim.binding.integrity.VcIntegrity is not visible from class loader

After logging into the vSphere Web Client, I found the following error in the Update Manager section:

 interface com.vmware.vim.binding.integrity.VcIntegrity is not visible from class loader

I started off by restarting the VMWare vSphere Update Manager Service for the affected vCSA:

1. Log into vCenter using the administrator@vsphere.local account.
2. Home - System Configuration - Services - Restart

This did not resolve my issue...  Unfortunately, I had to resort to something a bit more drastic and disruptive.  Since I did not know which service(s) needed to be restarted I restarted all of them.

SSH/Console into the affected server and run the following commands:

service-control --stop --all
service-control --start --all

This resolved the issue and I was able to see the proper output under the Update Manager tab.

vSphere 6 - Fix - Cannot complete operation due to concurrent modification by another operation.

Add more resources to a VM, not a problem right?  Well, not this time.  When attempting to add memory to a VM, I received the following error:

"Cannot complete operation due to concurrent modification by another operation."

I confirmed there were no other operations/modifications occurring on the VM at the same time.  In the past, I would just SSH/Console into the vCSA and restart the services with the following commands.  (Warning - This process is disruptive):

service-control --stop –all

service-control –start –all

In this case, the services restart did not resolve the issue.  On a whim, I created and deleted a snapshot on the VM.  This action "freed" up the VM and allowed me to add the additional resources.

In the future, I'm going start off with the "snapshot" route since it's less disruptive.

Hope this helps!

"RDP Inception"

That's when you have to RDP into one box to get to another box.  That is all...

vSphere 6.5 - I/O error accessing change tracking file

Recently, I had to migrate a set of VMs from one vCenter server to another.   I was able to successfully migrate the dev and staging VMs without a problem.  This process took 15 minutes TOPS including reboots.

Of course, when I went to power up the production VMs, I received the following error through the Web Client:

I/O error accessing change tracking file

This error was a bit deceiving, a quick look at the error showed a familiar error:

Cannot open the disk '/vmfs/volumes/ or one of the snapshot disks it depends on.

This error message pointed me directly to the vmdk with the error and I was able to perform the steps in the following KB to resolve this issue.

https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2009244

vSphere 6.5 - Platform Services Controller (PSC) – Recommended Topology, Restore and Design Considerations

I recently had to work with VMware on a PSC issue and thought I'd share the information collected.

***Update 09.07.17*** - For those of you who are performing new install or upgrading to vSphere 6.5, VMWare recently released a vSphere 6.5 Topology and Upgrade Planning Tool. It can be found here:

https://vspherecentral.vmware.com/path-finder

General Info:

1. What’s being replicated:
1. SSO
2. Tags
3. Custom Roles
4. Global Permissions
5. Licensing.
2. Below is a link to supported and deprecated topologies for vSphere 6.5.
1. https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2147672
3. PSCs are in a Multi-master Model.  
4. A replication agreement (Bidirectional) is created at the time the PSC is built/installed.  All additional replication agreements must be created manually. (Vdcrepadmin)
5. Replication between PSCs occurs every 30 seconds.
6. A vCenter server can only point to one PSC.
7. Starting 6.5 U1, a single PSC can manage up to 15 vCenter servers.
8. 6.5 U1 supports up to 10 external PSCs in an SSO domain.
9. An SSO Site is a logical boundary.
10. Starting with version 6.5, you cannot re-point a vCenter server across SSO sites.  This WAS previously possible in version 6.0.
11. Enhanced Linked Mode requires an External PSC. 

Recommended PSC Topology:

1. Create as few replication agreements as possible, while preventing PSC Isolation.
2. A Ring Topology is recommended. (Linear)
   

Restore Information:

1. In the event that all your PSC instances have failed, and you have no good backups, you will need to REINSTALL your entire environment.
2. vCenter Server and PSC Restore Work Flow information.
1. https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.install.doc/GUID-1EE61D5C-EDE5-4D19-8F9C-520BE65248E1.html
3. Restoring a vCenter Server Environment with multiple PSCs:
1. Restore a Single Failed PSC.
1. Deploy a New PSC instance and join it to an active PSC in the same vCenter SSO domain and site.
2. https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.install.doc/GUID-2BECB679-280B-43BB-A3F4-E9BFDD281F47.html

2. Restore All Failed PSC Instances:
1. Restore the Most Recently Backed Up PSC VM.
2. Run the vcenter-restore Script registered with the restored PSC.
3. Deploy Additional PSC instances and join them to the same vCenter SSO domain.
4. Repoint back the connections between vCenter and PSC Instances.
5. https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.install.doc/GUID-7223A0A9-9E3C-4093-8121-9BD8B3DB793F.html

Important Commands:

1. vdcrepadmin – The command is located in the following directory of the PSC: /usr/lib/vmware-vmdir/bin.  KB 2127057
1. Showservers – Displays all the PSCs in a vSphere domain.
1. Ex. vdcrepadmin -f showservers -h psc1.vmware.local -u administrator -w VMw@re123
2. Showpartners – Displays the current partnerships from a single PSC within a vSphere Domain
1. Ex. vdcrepadmin -f showpartners -h psc1.vmware.local -u administrator -w VMw@re123
3. Showpartnerstatus – Displays the current replication status of a PSC and any of the replication partners of the PSC
1. Ex. vdcrepadmin -f showpartnerstatus -h localhost -u administrator -w VMw@re123
4. Createagreement and removeagreement – Allows for the creation and removal of additional replication agreements between PSCs in a vSphere domain.
1. Ex. vdcrepadmin -f createagreement -2 -h psc1.vmware.local -H psc4.vmware.local -u Administrator -w VMw@re123
2. vdcrepadmin -f removeagreement -2 -h psc1.vmware.local -H psc3.vmware.local -u Administrator -w VMw@re123
2. cmsso-util – Command used to repoint between external PSC within a site.  KB2113917.
1. Ex. cmsso-util repoint --repoint-psc systemname_of_second_PSC