Member Avatar for Member #129437

Hi,

I've been encountering a strange but annoying issue in my office in work - if anyone could shed any light on this or have any ideas as to what could be the cause it would be great.

We have 30 Dell PowerEdge blade servers with 2x NICs in each - these are running about 3-4 virtual machines at a time. Most of these 60 interfaces are connected to a Netgear GS748T(v3) 48 port managed switch. Last week a colleague complained that accessing his server was extremely slow. I noticed the NIC activity LED was flashing very rapidly and restarting the interface didn't do much. Moving it to a different port on the Netgear did resolve the issue and moving a server from a "good" port to a "bad" port showed that the problem stayed with the port on the Netgear.

More people are complaining and right enough, this issue is now effecting 7 or 8 ports at a time, all completely random. Hard or soft rebooting the Netgear switch will shift the problem randomly to different ports but never resolve it.

Today, I port mirrored one of the "bad" ports to see if there was a lot of traffic on it but there was nothing out of the ordinary - just regular network traffic, 90% of which was going to the Dell server rather than from. I then isolated the port to a VLAN so just my laptop and the Dell NIC were on it - the activity LED continued to blink like it was transferring at 100Mbps but Wireshark showed practically no traffic.

I did a factory reset on the Netgear but it booted back up with 7 servers suffering the same issue (and the web interface not loading at all :( ). It's a bit of a mystery because our traffic wouldn't be very heavy at all, certainly extremely well below the quoted maximum throughput of 40Gbps that Netgear state this model can cope with. I've also noticed for some time that some ports don't negotiate 1Gbps despite all the Dells' NICs supporting this speed.

It would be great to hear any suggestions, ideas or clues as to what the cause could be or ways to pinpoint it.

Thanks in advance! :)

  • 2 Contributors
  • 3 Replies
  • 215 Views
  • 1 Day Discussion Span
  • Latest Post Latest Post by Member #129437

Dani AI

Generated

Useful follow‑up and practical checklist for the symptom of a NIC LED flashing furiously while packet captures show almost nothing. As demonstrated, the fault can move between ports and can be invisible to a simple Wireshark capture — which points away from a single server NIC fault and toward L1/L2 anomalies, inline appliances, or switch/firmware oddities. ’s point about vendor reliability is valid, but real troubleshooting steps are cheaper and faster than throwing hardware away.

Stepwise approach to reproduce and isolate the problem

  1. Verify whether the problem follows the server or the switch port by swapping the server to a known-good port and moving a good server into the suspect port.
  2. Swap patch cables and SFPs. Bad cables or transceivers can cause link-pulse activity without visible payloads.
  3. Check switch per-port counters: CRC/input errors, collisions, drops, and link up/down events. Look for MAC flapping or unexpectedly large MAC tables (flooding).
  4. Use a dedicated capture host attached to a mirror port; disable NIC offloads on the capture host (GRO/GSO/TSO) so the capture shows what’s on the wire, then record with tcpdump. Example: 
    sudo ethtool -K eth0 gro off gso off tso off
sudo tcpdump -i eth0 -w mirror.pcap
  5. Inspect spanning tree and BPDU activity to detect loops. If you suspect inline monitoring hardware, temporarily remove or bypass it — inline probes/taps that fail or are misconfigured often create loops, floods, or duplicate MACs.

Mitigation and prevention

  • Enable portfast/BPDU guard on access ports and storm-control thresholds for broadcast/multicast/unknown‑unicast.
  • Configure errdisable recovery for common fault types.
  • Keep switch firmware current and check vendor release notes for known bugs on the GS748T v3.
  • Prefer passive taps or properly configured packet brokers for monitoring; avoid placing opaque inline devices on production links.

If the issue recurs after these checks, collect captures and switch logs and engage vendor support — they can help interpret ASIC-level counters that aren’t visible from the CLI.

Recommended Answers

All 3 Replies

Member Avatar for OldTime

I've had several NetGear switches have this problem with ports going bad. NetGear's solution? "Use another port". My solution? Don't use NetGear switches if you can afford something more reliable, such as Cisco.

Member Avatar for Member #129437

I've had several NetGear switches have this problem with ports going bad. NetGear's solution? "Use another port". My solution? Don't use NetGear switches if you can afford something more reliable, such as Cisco.

Although I don't dispute Negear's reliability - I can't conclude a hardware issue yet when soft reboots move the problematic ports around. Besides, we're a small company and those switches were there before me - I couldn't get rid of them if I tried. Managed switches 'aint cheap! :(

Member Avatar for Member #129437

Discovered the issue - bizarrely, it was a network monitoring device - once we unplugged it, all was normal. We've contacted the manufacturer who are meant to be one of our partners about WTF was happening.

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.