vSphere 6.5 - I/O error accessing change tracking file

Recently, I had to migrate a set of VMs from one vCenter server to another.   I was able to successfully migrate the dev and staging VMs without a problem.  This process took 15 minutes TOPS including reboots.

Of course, when I went to power up the production VMs, I received the following error through the Web Client:

I/O error accessing change tracking file

This error was a bit deceiving, a quick look at the error showed a familiar error:

Cannot open the disk '/vmfs/volumes/ or one of the snapshot disks it depends on.

This error message pointed me directly to the vmdk with the error and I was able to perform the steps in the following KB to resolve this issue.

https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2009244

vSphere 6.5 - Platform Services Controller (PSC) – Recommended Topology, Restore and Design Considerations

I recently had to work with VMware on a PSC issue and thought I'd share the information collected.

***Update 09.07.17*** - For those of you who are performing new install or upgrading to vSphere 6.5, VMWare recently released a vSphere 6.5 Topology and Upgrade Planning Tool. It can be found here:

https://vspherecentral.vmware.com/path-finder

General Info:

1. What’s being replicated:
1. SSO
2. Tags
3. Custom Roles
4. Global Permissions
5. Licensing.
2. Below is a link to supported and deprecated topologies for vSphere 6.5.
1. https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2147672
3. PSCs are in a Multi-master Model.  
4. A replication agreement (Bidirectional) is created at the time the PSC is built/installed.  All additional replication agreements must be created manually. (Vdcrepadmin)
5. Replication between PSCs occurs every 30 seconds.
6. A vCenter server can only point to one PSC.
7. Starting 6.5 U1, a single PSC can manage up to 15 vCenter servers.
8. 6.5 U1 supports up to 10 external PSCs in an SSO domain.
9. An SSO Site is a logical boundary.
10. Starting with version 6.5, you cannot re-point a vCenter server across SSO sites.  This WAS previously possible in version 6.0.
11. Enhanced Linked Mode requires an External PSC. 

Recommended PSC Topology:

1. Create as few replication agreements as possible, while preventing PSC Isolation.
2. A Ring Topology is recommended. (Linear)
   

Restore Information:

1. In the event that all your PSC instances have failed, and you have no good backups, you will need to REINSTALL your entire environment.
2. vCenter Server and PSC Restore Work Flow information.
1. https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.install.doc/GUID-1EE61D5C-EDE5-4D19-8F9C-520BE65248E1.html
3. Restoring a vCenter Server Environment with multiple PSCs:
1. Restore a Single Failed PSC.
1. Deploy a New PSC instance and join it to an active PSC in the same vCenter SSO domain and site.
2. https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.install.doc/GUID-2BECB679-280B-43BB-A3F4-E9BFDD281F47.html

2. Restore All Failed PSC Instances:
1. Restore the Most Recently Backed Up PSC VM.
2. Run the vcenter-restore Script registered with the restored PSC.
3. Deploy Additional PSC instances and join them to the same vCenter SSO domain.
4. Repoint back the connections between vCenter and PSC Instances.
5. https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.install.doc/GUID-7223A0A9-9E3C-4093-8121-9BD8B3DB793F.html

Important Commands:

1. vdcrepadmin – The command is located in the following directory of the PSC: /usr/lib/vmware-vmdir/bin.  KB 2127057
1. Showservers – Displays all the PSCs in a vSphere domain.
1. Ex. vdcrepadmin -f showservers -h psc1.vmware.local -u administrator -w VMw@re123
2. Showpartners – Displays the current partnerships from a single PSC within a vSphere Domain
1. Ex. vdcrepadmin -f showpartners -h psc1.vmware.local -u administrator -w VMw@re123
3. Showpartnerstatus – Displays the current replication status of a PSC and any of the replication partners of the PSC
1. Ex. vdcrepadmin -f showpartnerstatus -h localhost -u administrator -w VMw@re123
4. Createagreement and removeagreement – Allows for the creation and removal of additional replication agreements between PSCs in a vSphere domain.
1. Ex. vdcrepadmin -f createagreement -2 -h psc1.vmware.local -H psc4.vmware.local -u Administrator -w VMw@re123
2. vdcrepadmin -f removeagreement -2 -h psc1.vmware.local -H psc3.vmware.local -u Administrator -w VMw@re123
2. cmsso-util – Command used to repoint between external PSC within a site.  KB2113917.
1. Ex. cmsso-util repoint --repoint-psc systemname_of_second_PSC