Upgrading my Dell PowerEdge R820 with an NVMe SSD!

 This was the best $14 I've spent on my Homelab.   

The processor and memory specs on my Dell PowerEdge R820 running VMWare ESXi 7 are completely fine. Unfortunately, the storage performance was lacking... So, I used a generic $14 PCIe card that allowed me to use an existing M.2 SSD drive as a VMWare datastore. IOPs went from 1K to 54K and power consumption went down by 40W.

Quick video on the process:

https://www.youtube.com/watch?v=fG4WQbLUhuE

Azure Migrate - How to reset the Azure Migrate Appliance

One of the limitations of Azure Migrate is that each Azure Migrate Appliance can only be associated with one Migrate project.  

After successfully migrating a VM from on-prem to Azure, here’s a quick and easy way to reset the appliance.

 

1. Open the following JSON file:

C:\ProgramData\Microsoft Azure\Config\appliance.json

Update "IsApplianceRegistered" from "True" to “False” and save the JSON file.

 

 

2. Restart the server.  Upon restart, the appliance has now been reset.

3.  That's it!  The appliance is now ready to be used for your next migration project. 

vCSA 7 Update Failed - "Error in method invocation [Errno 2] No such file or directory" and "Update installation is in progress"

Well, that didn't go as planned...

I had a VMWare vCSA that was running version 7.0 GA that was in need of an update.  The pre-update check ran successfully, the update was staged (5GB) and I began the installation.  

After an hour, the progress was still 0%, I clicked "Cancel Installation" and received the following error:

Error in method invocation [Errno 2] No such file or directory

I then performed a graceful restart of the appliance and received the following from the VAMI when logging in using root creds:

"Update installation is in progress"

To resolve this issue I SSH'd into the appliance and performed the following:

1. Create a copy of the software_update_state.conf file.  The file is located in the /etc/applmgmt/appliance/ directory

cp software_update_state.conf software_update_state_backup.conf

2. Stop the applmgmt service.

service-control --stop applmgmt

3. Delete the software_update_state.conf file.

rm software_update_state.conf

4. Confirm the applmgmt service starts successfully.

service-control --start applmgmt

I was then able to successfully log into the VAMI and complete the install. 

5. I un-staged the update. (Neat, reboot not required)

6. This time, I selected "Stage and Install"

7. The update installed successfully and the vCSA is now running the latest release. 

This appears to be an issue when a vCSA update is unsuccessful.  Deleting the original software update state config file and having the vCSA create a fresh one looks to resolve the issue.

Deploy OVF Template - A Required Disk Image Was Missing

I was asked to look into an OVF import issue.  He was receiving the following error:  "A required disk image was missing."

I looked in the .ovf file and confirmed that the names of the .vmdk and .ISO files referenced matched the files selected during the deployment process.  (I've had instances where the files were renamed.  Why, I don't know).  Everything checked out.  

I was using the HTML 5 client as recommended.  On a whim, I tested the process using the old school Flash client.  The same files and process was used.  Using the Flash Client, the job completed successfully.

Unexpected, but it worked. 

vCSA - 503 Service Unavailable - Failed to connect to endpoint

I was recently asked to look into a vCSA issue. The user was unable to log in using the web client. The vCSA was subsequently rebooted and it then produced the following error: “503 Service Unavailable”


503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http20NamedPipeServiceSpecE:0x00007fb7d00200a0] _serverNamespace = / action = Allow _pipeName =/var/run/vmware/vpxd-webserver-pipe

Upon running “service-control –status” in the vCSA, it was determined that  the vapi-endpoint service was in a stopped state.


It tuns out The Security Token Service (STS) Certificate had expired. If your vCSA was deployed using version 6.5 U2 or later, the STS cert may only be good for 2 years.


Sadly, there is no warning or easy way to determine when the STS Cert will expire using the HTML 5 client.


Resolution:
Part 1:
VMware has created a script to resolve this issue (fixsts.sh).  It can be found here:

https://kb.vmware.com/s/article/76719


1. Create a snapshots of all vCSAs (and external PSCs, if applicable).
2. Copy the .sh script to the /tmp dir of the appliance running the PSC role.
3. Make the script executable by running “chmod +x fixsts.sh”
4. Execute the file “./fixsts.sh”
5. Restart the appliance and confirm functionality.
6. The script only needs to be run once per SSO Domain.

Part 2:
Upon restart, the Web client was then throwing an Error 400. The user certificates needed to be replaced as well.

1. Launch the Certificate Manager utility: /usr/lib/vmware-vmca/bin/certificate-manager

2. Select Option 6.

3. The default options were taken except for the following:

Enter proper value for 'IPAddress' [optional] : IPADDRESS

Enter proper value for 'Hostname' [Enter valid Fully Qualified Domain Name(FQDN), For Example : example.domain.com] : VCFQDN

Enter proper value for VMCA 'Name' : VCSHORTNAME

4. The Certificate Replacement process was stopped at 85% by hitting CTRL+C, and the services were manually started by running “Service-control –start –all”.

The process was stopped at 85% since the certificate replacement was completed. At 85%,, the vCSA was just waiting for all the processes to restart. If any of the services fail to start, the certificate replacement will be rolled back.

5. This process can be done at any time. It only affects the vCSA the process is performed on.


Checking the Expiration Date of the STS Certificate: 

Method 1 - Flash Client - login using administrator@vsphere.local using the:

Method 2
1. Download the “checkSTS.py” script from the following location: https://kb.vmware.com/s/article/79248
2. Copy the script the /tmp dir.
3. Run the Python Script “python checksts.ph”

Repairing Mego Buck Rogers Action Figures

During the Shelter In Place, I've been cleaning up the house and going through my childhood "Treasures".  

A bit of background - After Star Wars, there were a bunch of sci fi TV shows and movies like Alien, Battlestar Galactica, The Black Hole and off course Buck Rogers.

The "Buck Rogers in the 25th Century" TV Series ran for 2 seasons starting in 1979.  As a 7 year old I absolutely loved this show.  I totally had a crush on Erin Gray who played colonel Wilma Deering on the show.

All my Star Wars action figures were still intact, but my Buck Rogers action figures didn't fare as well.  It appears that over time, the rubber band holding the figure together begins to dry rot and snap. 

It's a quick fix,  just an inexpensive O-Ring and a single Philips head screw. 

Twiki used a #7 O-Ring

O.D    1/2"

I.D.     3/8"

Wall    1/16"

Buck Rogers used a #11 O-Ring

O.D    3.4"

I.D.     9/16"

Wall    3/32"

vSphere 7 Lab on a Dell Poweredge R820

With the Shelter in Place Order, I took the opportunity to fire up the ol' Home Lab.   Nothing fancy,  I affectionately call it my "Ghetto Lab"

The timing is perfect,  I'm stuck at home and vSphere 7 just went GA this week.  I'm taking this opportunity to do some testing and play with the latest release of vSphere from VMware. 

The core of my lab consists of a pair of Dell PowerEdge R820s.   Sadly, this model hasn't been officially supported by VMware since ESXi 6.5.  Fortunately for me, it runs fine in my lab for testing!  Below are the server specs for those who would also like to test on older hardware.

Server Specs:
Dell PowerEdge R820
Intel Xeon E5 - 4620
Broadcom PERC H710
Qlogic BCM57800 1GB Ethernet Adapter

Stay positive, use this time to learn and better yourself.  Happy Testing!