Compellent VMware - Slow add of Thick Provision Eager Zeroed Hard Disks.

One of our VMware environments uses the Compellent SC8000 for it's storage.  We noticed that over time,  the time it takes to add a Thick Provision Eager Zeroed Hard Disks would drastically increase.

This greatly affected the creation of new VMs, adding of hard disks and VM restores using Commvault.

Some Numbers...
On a newly created datastore, adding a 25GB drive to an existing VM took roughly 20 seconds.

That same VM was storage vMotioned onto an existing datastore and the process of adding a 25GB Hard disk took 30 MINUTES.  This datastore had plenty of free space to accomodate the VM and the newly added drive.

To resolve this issue, use the Thin Provisioning Block Storage reclamation (UNMAP) primative.  This process allows deleted blocks to be reclaimed on thin provisioned LUNS that support the VAAI UNMAP primative. This process is manual and must be run in each Datastore.

1.  SSH into an ESXi host that is attached to the Datastore you wish to run the SCSI UNMAP procedure on.

2. Drill down to the datastore and run the vmkfstools -y command.  The value at the end is the percentage of deleted blocks to reclaim.  In this example, I'm reclaiming 50%.

Although the process takes only 30 seconds to complete for a 1TB datastore,  I would recommend performing this task after hours.

vmkfstools -y 50

3. As a safe measure, the command was run a second time on the same datastore.  This time specifying 60%.

Note: This command has been deprecated and replaced by esxcli storage vmfs unmap in ESXi 5.5. 

Hard Disk creations and additions are now back to their normal speed.

Here are some before and after numbers as seen from the Compellent side:

 

References:

 http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2014849

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2057513