Wednesday, July 18, 2018

Unable to access file since it is locked - An error occurred while consolidating disks: msg.fileio.lock.

This is an oldie but a goodie.  

I was recently asked by the backup team to look into an error they were seeing in NetBackup.  The error was that NetBackup was unable to consolidate a virtual machine's disks.



Just right click on the VM --> Snapshot --> Consolidate.  Done!  Not so fast, this time I received the following error:

An error occurred while consolidating disks: msg.fileio.lock.


Unfortunately,  the usual create/delete snapshot and vMotion of VM did not work.

To resolve this issue perform the following:

1. SSH or Console into the ESXi host.
2. View the vmware.log file of the offending VM and look for the locked file:

ex.  /vmfs/volumes/offendingVM/offendingVM-dir/vmware.log 



3. Run the vmkfstools -D command against the locked .vmdk to determine the MAC address of the ESXi host which has the lock.  The MAC address of the ESXi host which has locked the file is circled in RED.


4. Log into the vCenter server using you're favorite client, then look for the ESXi host which has the NIC matching the MAC above.

5. Place the host locking the vmdk in Maintenance Mode.  Then restart the hostd service:

/etc/init.d/hostd restart


6.  Exit Maintenance Mode.  I was then able to successfully perform the Consolidate function.  As an additional test, I created and deleted a test snapshot.

Tuesday, July 10, 2018

VMware vCSA VAMI :5480 - Certificate Error - Not Secure - You cannot visit right now because the website uses HSTS.

I recently replaced the self signed cert on our vCSA with one generated from a proper CA server.  The Web Client and vSphere (HTML5) client showed the nice green Secure padlock.

However, when I tried to access the VMware Appliance Management Interface (VAMI), I received a Not Secure prompt.  I was unable to proceed to the site. 



Advanced details showed the following;


MyServer.MyDomain.com normally uses encryption to protect your information. When Google Chrome tried to connect to MyServer.MyDomain.com this time, the website sent back unusual and incorrect credentials. This may happen when an attacker is trying to pretend to be MyServer.MyDomain.com, or a Wi-Fi sign-in screen has interrupted the connection. Your information is still secure because Google Chrome stopped the connection before any data was exchanged.
You cannot visit MyServer.MyDomain.com right now because the website uses HSTS. Network errors and attacks are usually temporary, so this page will probably work later.

Well, there appears to be a bug where with the VMware vCSA 6.X.  After applying a new vCSA certificate, the VMWare Appliance Management Interface (VAMI) does not display the new certificate.

To resolve this issue for a vCSA running 6.5,  perform the following:

1. Copy the CA cert to the following directory : /etc/applmgmt/appliance/ca.crt

By default, the ca.crt file does not exist in this directory.  FYI, Applmgmt is the VMware Appliance Management Service.



2. Using VI, open the following file: /opt/vmware/etc/lighttpd/lighttpd.conf

3. Add the following line to the file: 
ssl.ca-file="/etc/applmgmt/appliance/ca.crt"


3. Restart the VAMI Service by running: /etc/init.d/vami-lighttp restart



Enjoy the nice Green Secure lock!