How to Shutdown a Cisco UCS Central VM

We recently had to restart the Cisco UCS Central VM (ver. 1.5).  I couldn't find a way through the Web Management Portal/GUI, so a Shutdown was performed through the console.

1. As a precaution, create a VMware snapshot.
2. SSH into the Cisco UCS Central VM using the admin credentials.
3. Then, switch the context to local-mgmt:

4. Below are the options available within this context.  Note - shutdown and restart.

5. Enter the shutdown (or restart) option.  Enter yes to confirm.

Nissan Vmotion 2.0 - VMWare vMotion

As a user of VMware products for over a decade, the following article made me chuckle.

http://www.caranddriver.com/news/nissan-vmotion-20-concept-photos-and-info-news

My co-worker said the Nissan Vans should be called "Storage Vmotion".  Har har.

vMotion Error: A general system error occurred: The source detected that the destination failed to resume.

This past weekend, a VM had unexpectedly powered off (Of course on Jan 1st, Yay me! ).  I was able to successfully power on the VM and confirmed the application was up and running.

However, I noticed that a vMotion that was kicked off by VMTurbo/Turbonomics had failed. (I'm still getting used to their new name).

The error was:

The VM failed to resume on the destination during early power on.

Module DiskEarly power on failed.

Cannot open the disk '/vmfs/volumes/myvm/myvm/myvm.vmdk' or one of the snapshot disks it depends on.

Could not open/create change tracking file

This was similar to the following KB, but in my case the VM was powered on.  For completeness, I also did not have any snapshot/delta files for the VM.

https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2009244

Unfortunately, I had to coordinate the down time to resolve this issue.  Once I received the approval, here are the steps I took:

1. Gracefully shut down the VM.
2. Note the offending .vmdk file in the error message.  Capture the SCSI Controller type AND the Virtual Device Node of the offending .vmdk/Hard Disk.

3. SSH in the host and drill down to the folder containing the VM.  Once again, I did not have any snapshot files for this particular VM.
4. Use the Move (mv) command to rename the ctk file of the offending vmdk.
     ex. mv myvm-ctk.vmdk myvm-ctkold.vmdk.
5. Re-add the disk to the VM using the appropriate Virtual Device Node using the vSphere Client.
6. Edit the offending .vmdk descriptor file by commenting out any references that specify the use of a ctk file.

7. Power on VM.
8. Test VM functionality.  vMotion, creating/deleting snapshots etc.

Apparently the cause cause of this issue is when the filename-ctk.vmdk file is not cleared.  No reason was given as to what caused this in the first place.

Cisco UCS Central: Manual Refresh of Inventory.

We recently had an issue where the power state of several B-Class servers did not match within the server and the service profile view.

The Server view correctly had the power state as Power On.  However, the service profile associated with this server incorrectly showed the Status as Power Off.  There was definitely a disconnect.  Shutting down and powering on the server did not resolve the issue.

To resolve this issue, ssh in the Cisco UCS Central VM:

Once logged in enter the folowing:
connect resource-mgr
scope domain-mgmt
show ucs-domain detail

The last command will give you the ID of the system you would like to perform the refresh on:

Confirm that you are scope'd in on the correct domain by running the following;
scope ucs-domain xxxx
show detail

Once you have confirmed you are in the correct domain, run:
refresh-inventory
commit-buffer

The Commit-buffer command commits the transaction.

 Cisco UCS Central is now syn'd up from Central down to the servers.

Windows Server 2016 pvscsi and vmxnet3 drivers

With the release of the GA version of Windows Server 2016, it's time to start playing!

I spun up a fresh VM using the Paravirtual SCSI Controller (PVSCSI) and a vmxnet3 NIC.  Well, as expected, the PVSCSI controller driver was not included on the Windows Server 2016 ISO.  For this initial VM build, I simply supplied the SCSI controller driver, and once I installed the VMWare tools, the vmxnet3 nic was recognized.

However, I wanted to provide the SCCM team with the necessary VMware drivers to successfully deploy a 2k16 server within VMware if the need arises.

Here are the steps to get the necessary drivers:

Download the appropriate version of VMware-tools-xxx-xxx-x86_64.exe for your environment from the following location:

https://packages.vmware.com/tools/esx/index.html

Then run the executable with the /a option.  This will allow you to extract all the driver files.

 Select Next.  Then specify the location you would like the drivers and click Install:

 The drivers will be located in the location specified above:

Now you can provide the SCCM team with the necessary drivers and have them inject them into the WinPE Boot Media. 

How to test Jumbo Frames on a vmkernel port

Here's an "Oldie but a Goodie".  I've been using it recently to sort out some iSCSI issues, so I figured I'd share.

The vmkping command sources a ping from a vmkernel port on the ESXi host.

-I - Allows you to specify a vmkernel port.

-d - Sets the Do Not Fragment bit on the packet.  This MUST be used to test Jumbo Frames.

-s - is the size of the payload.  

From the ESXi shell run the following command:

vmkping -I vmk1 -d -s 8972 x.x.x.x

In my case, I specified vmk1, which is used for the software iSCSI initiator.  x.x.x.x. was my storage array. 

The payload size must be set to 8972 due to the ICMP (8 bytes) and IP (20 bytes) header. .

Fix - Cisco UCS - Virtual Media redirection is already in use by another user

While performing an ESXi install on one of the Cisco UCS Blades, I received the following error:

I confirmed that no other users were using the UCS KVM, rebooted the blade from within the ESXi OS and through UCS Central etc.

To resolve this issues log into UCS Manger.  Then from the Equipment tab select the server exhibiting this behavior:

From the Servers General tab, select Recover Server and select Reset CIMC.  Resetting the CIMC will not cause a disruption of service.

That's it, I was then able to launch the KVM and attach Virtual Media.