ICX switches have an icmp response time with significant delay

[kshal]

Hi everybody, we have some strange behavior on Ruckus ICX switches.
If we try to ping core switch (stack of two ICX 7750) from user PC, or just from access switch stack (ICX 7250) response time can be significantly varies.
It could be 150ms, or 500ms, or some icmp packets could be without any response.
We suppose that it could be STP Topology Changes and ARP table flush, so we switched off links to another distributed switches to isolate problem.
After that we switched off secondary link from access switches to core, it doesn't help.
Finally, we leave only one active connection core-access, but icmp response time still have a jitter.
Now we have no idea how to isolate a problem.
Switch firmware was changed too, now it is 08.0.95bc (was 08.0.90dT203).
Switch utilization is 1%, uplink utilization is 1 to 3%, packets dropping are absent.
Logging buffer has no anomalies.
Network topology is a star with dual links to core from access switches stack.
Uplinks are L2 LACP.

sh cpu-utilization                
1 percent busy, from 184 sec ago
1   sec avg:  1 percent busy
5   sec avg:  1 percent busy
60  sec avg:  1 percent busy
300 sec avg:  1 percent busy

ping 172.16.12.1 count 60 interval 1
Sending 60, 16-byte ICMP Echo to 172.16.12.1, timeout 5000 msec, TTL 64
Type Control-c to abort
Reply from 172.16.12.1     : bytes=16 time=1ms TTL=64
Reply from 172.16.12.1     : bytes=16 time=3ms TTL=64
Reply from 172.16.12.1     : bytes=16 time=1ms TTL=64
Reply from 172.16.12.1     : bytes=16 time=9ms TTL=64
Reply from 172.16.12.1     : bytes=16 time=1ms TTL=64
Reply from 172.16.12.1     : bytes=16 time=3ms TTL=64
Reply from 172.16.12.1     : bytes=16 time=62ms TTL=64
Reply from 172.16.12.1     : bytes=16 time=3ms TTL=64
Reply from 172.16.12.1     : bytes=16 time=1ms TTL=64
Reply from 172.16.12.1     : bytes=16 time=1ms TTL=64
Reply from 172.16.12.1     : bytes=16 time=1ms TTL=64
Reply from 172.16.12.1     : bytes=16 time=157ms TTL=64
Reply from 172.16.12.1     : bytes=16 time=1ms TTL=64
Reply from 172.16.12.1     : bytes=16 time=1ms TTL=64
Reply from 172.16.12.1     : bytes=16 time=1ms TTL=64
Reply from 172.16.12.1     : bytes=16 time=1ms TTL=64
Only the last 16 ping replies were printed.
Success rate is 100 percent (60/60), round-trip min/avg/max=1/6/157 ms.

[sunny_gupta]

Please note that RUCKUS Fastiron code which runs on ICX treats ICMP traffic directed to the switch as low priority since it can impact the CPU.  The better test case would be to ping through the switch rather than to the switch in order to get a good view of latency in the network.

Please try to ping a device through the switch and let us know if that is also experiencing any high latency.

[kshal]

@sandeep_gupta_89lya7d7doyxy 

Yes, we tried to do that earlier, and icmp delay increased synchronously on different VLANs.
And icmp packet loss is synchronously too.

[sunny_gupta]

If you are still experiencing loss/latency then I would check the links and ensure there are no errors.  If fibre, clean the glass, if copper confirm the cabling is within spec and passed.

There is no reason to have a direct connection through a switch with high latency unless the end devices are the culprit.  ICX switching is line-rate and wire-speed especially in a low-utilized environment.

If you check all these and still have issues, do open a support ticket with RUCKUS TAC to investigate futher/debug.

[kshal]

We have two networks - CCTV and LAN, both on ICX-7750, and both have this issue.

I'm not sure that approx hundred uplinks are dirty, or failed.

All counters on physical interfaces are looking good (statistic on most loaded interface is attached below).

Is it any deep-dive diagnostic command to detect HW problem, packet dropping etc?

 Port 1/1/31 Counters:
         InOctets          72871300057           OutOctets         103956128617
           InPkts            122940850             OutPkts            131590140
  InBroadcastPkts               174064    OutBroadcastPkts               175578
  InMulticastPkts              1482989    OutMulticastPkts                49591
    InUnicastPkts            121283798      OutUnicastPkts            131364971
        InBadPkts                    0
      InFragments                    0
       InDiscards                    0           OutErrors                    0
              CRC                    0          Collisions                    0
         InErrors                    0      LateCollisions                    0
      InGiantPkts                    0
      InShortPkts                    0
         InJabber                    0         OutDiscards                    0
   InFlowCtrlPkts                    0     OutFlowCtrlPkts                    0
     InBitsPerSec            138409264       OutBitsPerSec            196845056
     InPktsPerSec                29588       OutPktsPerSec                31538
    InUtilization               14.30%      OutUtilization               20.17%