High %DRPRX - Dropped Receive Packets - ESXTOP

We recently had an data center assessment performed and it showed a high number of %DRPRX instances in our virtual environment.

I ran through the following:

1. Confirmed that the CPU utilization was good at the Host and VM levels.
2. Confirmed that there was no memory swapping.
3. Confirmed that the VMs were using the VMXNet 3 Nic vs. the E1000.
4. Confirmed that updates outlined in VMware KB2052917 have been applied to all hosts. 

All of the above checked out clean in the environment.

Then I ran RESXTOP (Remote ESXTOP gives you real-time info of the host).  Sure enough, every few cycles, %DRPRX (Dropped packets received) would go from 0.00 to showing double digit numbers, then back down to 0.00…  I checked another ESXi host, and it exhibited the same behavior.

I then disabled the Virtual Distributed Switch (vDS) “Health Check” function, and that stopped the spikes in the dropped packets.  (Must be done through the Web Client)

 

It turns out, the Health Check uses periodic broadcasts during the Teaming, VLAN and MTU check process.

 The “Health Check” feature is disabled by default.  However, during the vDS setup process, I saw the Health Check option and figured  “What can it hurt?”.Well, now we know....

Moving forward,  I'll enable this feature temporarily after any configuration changes to confirm that Teaming, VLAN and MTU is setup correctly then immediately disable the feature.  (ex. VLAN changes, ESXi host replacements etc.)

VMware Online Launch Event - Summary

Just a Quick summary of todays event.  There were 4 major announcements:

1. VMware NSX and VMWare vCoud Air - Hybrid Networking Services - You can now leverage VMware NSX to extend your private data centers into vCoud Air.  NSX currently has over 400 paying cutomers.

2. VMware Integrated OpenStack (VIO) - This functionality is free to vSphere Enterprise Plus customers.

3. VMware vSphere 6 - over 650 new features:
    a. 2x more hosts per cluster.  4x more VMs per Host. 4x more Virtual RAM per VM.

    b. vMotion Across vCenter Servers.
    c. Long-Distance vMotions.  This feature requires <100ms of delay.
    d. Fault Tolerance for Multi-Proc VMs (up to 4 vCPUS)
    e. vCloud Air Disaster Recovery:

    f. Instant Clone for Big Data, Virtual Desktops and containers.

    g. Usability:

4. VMware vSAN 6 and Virtual Volumes (vVols):
    a. All flash architecture vs. hybrid design in previous versions.
    b. More snapshots. 32 vs. 2.
    c. 64 nodes and up to 7M IOPs.
    d. vSAN - Rack-Awareness for HA.  Uses fault domains.

    e. Broad support for vVols.

    f. vVols - VM-Aware storage

VMware Online Launch Event - Able to log in?

The launch event was scheduled to start at 1pm.  I was finally able to log in at 1:05pm.

As of 1:10pm I'm still seeing this?  Any updates?  Webcast issues?

 

Update: The webcast started shortly after this post was published.  Event summary to follow....

Compellent VMware - Slow add of Thick Provision Eager Zeroed Hard Disks.

One of our VMware environments uses the Compellent SC8000 for it's storage.  We noticed that over time,  the time it takes to add a Thick Provision Eager Zeroed Hard Disks would drastically increase.

This greatly affected the creation of new VMs, adding of hard disks and VM restores using Commvault.

Some Numbers...
On a newly created datastore, adding a 25GB drive to an existing VM took roughly 20 seconds.

That same VM was storage vMotioned onto an existing datastore and the process of adding a 25GB Hard disk took 30 MINUTES.  This datastore had plenty of free space to accomodate the VM and the newly added drive.

To resolve this issue, use the Thin Provisioning Block Storage reclamation (UNMAP) primative.  This process allows deleted blocks to be reclaimed on thin provisioned LUNS that support the VAAI UNMAP primative. This process is manual and must be run in each Datastore.

1.  SSH into an ESXi host that is attached to the Datastore you wish to run the SCSI UNMAP procedure on.

2. Drill down to the datastore and run the vmkfstools -y command.  The value at the end is the percentage of deleted blocks to reclaim.  In this example, I'm reclaiming 50%.

Although the process takes only 30 seconds to complete for a 1TB datastore,  I would recommend performing this task after hours.

vmkfstools -y 50

3. As a safe measure, the command was run a second time on the same datastore.  This time specifying 60%.

Note: This command has been deprecated and replaced by esxcli storage vmfs unmap in ESXi 5.5. 

Hard Disk creations and additions are now back to their normal speed.

Here are some before and after numbers as seen from the Compellent side:

 

References:

 http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2014849

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2057513

Fix: /lib/ld-linux.so.2: bad ELF interpreter: No such file or directory

I had a request for a Static MAC address for one of my VMs.  I've been using a little tool called vmmacgen since 2005 to randomly generate a static MAC for me.  I SCP'd the file to my fresh install of CentOS 7, extracted the .gz and received the following error when I tried to run it:

/lib/ld-linux.so.2: bad ELF interpreter: No such file or directory
 
Luckily this was a quick fix.  Just run the following command to install the proper components and you'll be able to run the app (No reboot necessary):

yum -y install glibc.i686 

 

 I was then able to successfully run the tool:

Passed! VCP5-DCV Delta Exam - Exam code VCP550D

***01.11.16 I don't know if VMWare is being generous or they're getting desperate...  Exam has been extended AGAIN until March 31th 2016***

***05.21.15 Update: Looks like the exam was extended YET AGAIN until Sept 30th 2015***

***03.12.15 Update: Has this exam been extended again until May 8th 2015?***

https://mylearn.vmware.com/mgrReg/plan.cfm?plan=51919&ui=www_cert

Back in March 2014, VMware announced that their certifications will begin having expiration dates.  Existing VCPs have the option of taking the VCP5-DCV Delta exam to recertify.

 On December 11th, I received 4 emails stating my VCPs for ESX Server 2, VI 3, VCP 4, and VCP 5 were expiring in 3 months.  This email was the kick in the pants that I needed to begin studying for the Delta Exam.

Here are the steps I took:

1. Request authorization to retake the exam.  This process took about an hour.
 https://mylearn.vmware.com/mgrReg/login.cfm?ui=www_cert&RedirectURL=https%3A%2F%2Fmylearn%2Evmware%2Ecom%2FmgrSurvey%2FquizExtVerify%2Ecfm%3Fcourse%3D239201%26ui%3Dwww%5Fcert&sslRedirect=1

2.  Download the Exam Blueprint. Yeah, this was useless....  it pretty much included EVERYTHING!

3.  Participated in the free 1 hour What's New Fundamentals V5.5 course.
https://mylearn.vmware.com/mgrreg/courses.cfm?ui=www_edu&a=one&id_subject=51301

4.  I built up a fresh vSphere 5.5 lab using the vCenter Appliance.  My co-workers and I lovingly call it our "Ghetto Lab".  Nothing fancy but it get the basics done.

5.  When you register for the exam through Pearson Vue, you have 2 days to start and complete the exam.  The cost was a reasonable $120.

Although the exam details specify "new material" between 5.0/5.1 and 5.5, there were quite a few operational and general questions regarding vSphere 5.  Alot of the notes I took back in 2012 for the 5.1 exam were relevant for this delta exam.   That said, for my studies, I focused on the following new 5.5 features - vSAN, vSphere Flash Read Cache, vCOPS/vRealize, vSphere Replication and vDP.

My transcript was promptly updated the following day.

If you're a current VCP5-DCV, you have through March 10, 2015 to take the exam.

It's not too bad.  Study hard and Good luck!