How to troubleshoot or check network connection issues on Switches.

[jdryan]

Network connection Issues can be few of the things that can be a bit tricky to troubleshoot, Main reason being that there is always not one same cause behind the scenes that caused it, scenario to scenario the cause for the trouble can vary.   

Generally the Issues of not reaching the internet or not able to connect to the internet, not being able to connect to an internal/external resource and not able to get an ip address are seen to be synonymous.

Here being discussed are few of the most common ones that can be seen contribute to Issues of not reaching the internet or not able to connect to the internet, not being able to connect to an internal/external resource. 

For the Issues on IP addressing, please check here Check DHCP Issues here  

Few of the probable causes to name a few that can contribute to connection issues being discussed here are:

> DNS resolution not working
> Addressing based issues
> Gateway dropping connections / NATing issues
> Internal traffic suppression

.
> Check if DNS resolution is affected:
Generally, when the internet is not working, it implies the web pages are not loading on the browser.
Hence do check the below:
From a end device connected on the network:
Ping 1.1.1.1
Do a traceroute to 1.1.1.1 
Ping google.com / speed.cloudflare.com
And try to open google.com / speed.cloudflare.com

If the ping and traceroute works for 1.1.1.1 or 8.8.8.8
But not for google.com / speed.cloudflare.com

That would imply the DNS is not working / not resolving
• Here check if there are issues on the internal DNS systems
• If the public DNS systems is used then check on firewall/gateway to see if the same is being blocked or dropped
• Also check if there are security measures such as ACLs/policies in place that could be blocking them as well.

> Addressing based issues :
Check the IP address leased on the PC :
Check if the gateway can be reached
Do a "route print -a" or " route print " to confirm that right gateway is being populated on the PC

If all the above come up good, then :
Follow the ping tests to internal servers and external endpoints
If needed do a traceroute to check where its dropping.

Once the location of the drop is isolated, based on the role and function of the device, would need to look into what could be causing it.
• If on core : would need to check if there is any ACL blocking the response or no inter-network communication is taking place.
• If forwarded from core to gateway appliance and same is not aware where to forward the response back to would need to have that corrected.


>  Gateway connection Dropping :
This is rare, given the care and dligence taken during the config.
But if there were any policy change implemented that could drop traffic if the config is not done right.

>  Lastly, traffic suppression :
If in-case, there is a broadcast storm in the network this can happen.[as this is one of the most common causes] as a consequence.
Generally these are seen when there is a loop.
Depending on where the loop is and the traffic flow : the effect can be immediate or delayed.

Follow the below commands to check for traces of a loop.

In enable mode :
Clear statistics
Show statistics
Show statistics brief
Show statistics cpu-pkt
Show statistics ethernet
Show statistics lag
Show statistics management
Show statistics unit < specific member ID >

• This should help you trace where the frames are coming from, as generally when in this state.
The given tell-tale indicator would be the TX and RX of interfaces and their TX and RX utilization.

In case of a loop : for given 2 ports or more [ usually in pairs of 2 ] TX and RX becomes identical.
And RX is where we would need to start the trace from.

Note :
In case of Servers and Storage arrays, this can be seen as utilization spike. However this would be only on the ports that are designated for servers and storage arrays.
And in case of any Backup or read-write cycles : being in progress on the Storage array connected interfaces you will see the spike.
Hence when checking under this part : do keep the port layouts and the topology handy.

• Another indicator of loop would be MAC Movements, in this case it would be between a set of ports and the same would be high.
This helps confirm which ports are learning the same MAC address and between which ports the same is shifting across.

In config mode :
Router(config)#mac-movement notification threshold-rate
DECIMAL 1 to 50000 moves
Router(config)#mac-movement notification threshold-rate 20
sampling-interval Set the sampling interval
Router(config)#mac-movement notification threshold-rate 20 sampling-interval
DECIMAL 1 to 86400 seconds
Router(config)#mac-movement notification threshold-rate 20 sampling-interval 30
Router(config)#mac-notification interval
DECIMAL 1 to 3600 seconds
Router(config)#mac-notification interval 600

Check under show log for the records.

• In case due to the heavy traffic loads the CPU is getting overloaded and NMI is being triggered.
which is causing the device to reboot.

In this case above log entries and vital notifications will be missed.

To overcome this would suggest configuring a syslog server. As that can store the logs of the switch for review, should it re-start.
device(config)# configure terminal
device(config)# logging host <IP of the server >
device(config)# write memory
These above should help in isolating if not fixing the connection issues to internet to a point that would need attention.

These are a few things that can be done to check and isolate where the issue lies.