Odd Dell M620 NDC Issue

April 20, 2015  •  Leave a Comment

Earlier this year while troubleshooting a SAN issue Dell replaced all our Broadcom NDCs and mezzanine cards in our M620s with Intel X520s and excepting one DOA NDC everything was fine.  Don't know how much an impact the swap made on resolving the issue, but that is a story for another time.  A couple of weeks ago on one of the hosts though the vmnics associated with the NDC stopped showing up whether viewed through vCenter or the ESXi console.  Oddly they still showed up in the output of 'lspci -l', and in the CMC and iDRAC inventory views of the blade.  Since we weren't in a resource crunch I just put the host in maintenance mode and tried various troubleshooting steps as time allowed.

Reboot the host, no change.
Perform a virtual reseat of the blade, no change.
Remove the mezz cards and NDC, check connections and reinstall them, again no change.
Install the latest ixgbe driver from VMware, frustratingly still no change.

At this point I was about to reinstall the ESXi 5.5 U2 to rule that out before calling support; but on a whim I popped into the LifeCycle Controller and started poking around.  All the hardware diagnostics came back with green checkmarks as far as the eye could see, so I took a look at the configuration section for the NICs.  Everything looked normal until I got to the firmware properties page...  No firmware version was showing up on the NDC interfaces which seemed odd.  Took a look at the mezz cards to confirm that the field should be populated and 15.0.28 was clearly shown.

So I hop back into the CMC and head to the firmware update page, and see there that the current version field for the NDCs was empty only for this particular host.  At this point I've ruled out reinstalling the OS and decide to 'update' the firmware on the NDCs to 15.0.28 before throwing in the towel and calling support.  Once the update finished and the host rebooted all interfaces were showing the correct firmware version and all vmnics were once again showing up within ESXi. 


Comments

No comments posted.
Loading...

Archive
January (5) February March (1) April May June July August September October November (1) December (1)
January February March (1) April May June July August September October (1) November (4) December
January (1) February (1) March (2) April (1) May June July August September October November December
January February March April May June July August September October November December
January February March April May June July August September October November December