Network Connections Dropping On New vHosts

Within the last week I have replaced 4 old virtual hosts with new Dell R720s. Initially everything seemed to be working fine but this morning we started to see LUNs disappearing from the Storage tab and dropped connections to Management ports. We also experienced large amounts of latency when attempting to manage the environment.

After working with our storage supplier we determined that the problem was not on the Nexenta devices but elsewhere in the infrastructure.

After a lot of testing, searching and head-scratching I discovered that there is a known issue with the Broadcom network cards and ESXi 5.x. The R720s have Broadcom NetXtreme BCM5720 network cards and onboard connections.

The simple fix for this is to update the Dell pre-installed network drivers - in my case I went from 3.123c.v55.5 to the latest available version.

First I checked my current version by SSHing onto the vHost as root and running the commands;

lspci | grep BCM5720 (this displays all connected vmnics)

0000:01:00.0 Network controller: Broadcom Corporation NetXtreme BCM5720 Gigabit Ethernet [vmnic0]
0000:01:00.1 Network controller: Broadcom Corporation NetXtreme BCM5720 Gigabit Ethernet [vmnic1]
0000:02:00.0 Network controller: Broadcom Corporation NetXtreme BCM5720 Gigabit Ethernet [vmnic2]
0000:02:00.1 Network controller: Broadcom Corporation NetXtreme BCM5720 Gigabit Ethernet [vmnic3]

then;

ethtool -i vmnic0

After running ethtool you should get a response that looks like this;

driver: tg3
version: 3.123c.v55.5
firmware-version: FFV7.10.17 bc 5720-v1.34
bus-info: 0000:01:00.0

I then downloaded the new driver package from the VMware download centre at https://my.vmware.com/web/vmware/details?downloadGroup=DT-ESXI5X-BROADCOM-TG3-3137DV501&productId=285

There is a ReadMe document in the .zip download but for ease of reference I have reproduced it below;

COMMAND LINE INSTALLATION

New Installation
----------------

For new installs, you should perform the following steps:

1. Copy the VIB or offline bundle to the ESX server.  Technically, you can
           place the file anywhere that is accessible to the ESX console shell, 
           but for these instructions, we'll assume the location is in '/tmp'.

           Here's an example of using the Linux 'scp' utility to copy the file
           from a local system to an ESX server located at 10.10.10.10:
             scp VMware_bootbank_net-driver.1.1.0-1vmw.0.0.372183.vib root@10.10.10.10:/tmp

2. Issue the following command (full path to the file must be specified):
              esxcli software vib install -v {VIBFILE}
           or
              esxcli software vib install -d {OFFLINE_BUNDLE}
       
           In the example above, this would be:
              esxcli software vib install -v /tmp/VMware_bootbank_net-driver.1.1.0-1vmw.0.0.372183.vib

Note: Depending on the certificate used to sign the VIB, you may need to
      change the host acceptance level.  To do this, use the following command:
esxcli software acceptance set --level=<level>
      Also, depending on the type of VIB being installed, you may have to put
      ESX into maintenance mode.  This can be done through the VI Client, or by
      adding the '--maintenance-mode' option to the above esxcli command.


Upgrade Installation
--------------------

The upgrade process is similar to a new install, except the command that should
be issued is the following:

esxcli software vib update -v {VIBFILE}
or
esxcli software vib update -d {OFFLINE_BUNDLE}


VUM INSTALLATION

The VMware Update Manager (VUM) is a plugin for the Virtual Center Server
(vCenter Server).  You can use the VUM UI to install a VIB by importing
the associated offline bundle package (a ZIP file that contains the VIB and 
metadata).  You can then create an add-on baseline and remediate the
host(s) with this baseline.  Please see the vCenter Server documentation for

more details on VUM.

I used WinSCP to copy the .vib file to the /tmp/ location on the host. After upgrading you will need to reboot the host.

After following these steps, the network issues were resolved.