topic Re: ICX switches have an icmp response time with significant delay in Unleashed

ICX switches have an icmp response time with significant delay

kshal — Mon, 08 Feb 2021 09:21:00 GMT

Hi everybody, we have some strange behavior on Ruckus ICX switches.
If we try to ping core switch (stack of two ICX 7750) from user PC, or just from access switch stack (ICX 7250) response time can be significantly varies.
It could be 150ms, or 500ms, or some icmp packets could be without any response.
We suppose that it could be STP Topology Changes and ARP table flush, so we switched off links to another distributed switches to isolate problem.
After that we switched off secondary link from access switches to core, it doesn't help.
Finally, we leave only one active connection core-access, but icmp response time still have a jitter.
Now we have no idea how to isolate a problem.
Switch firmware was changed too, now it is 08.0.95bc (was 08.0.90dT203).
Switch utilization is 1%, uplink utilization is 1 to 3%, packets dropping are absent.
Logging buffer has no anomalies.
Network topology is a star with dual links to core from access switches stack.
Uplinks are L2 LACP.

sh cpu-utilization
1 percent busy, from 184 sec ago
1 sec avg: 1 percent busy
5 sec avg: 1 percent busy
60 sec avg: 1 percent busy
300 sec avg: 1 percent busy

ping 172.16.12.1 count 60 interval 1
Sending 60, 16-byte ICMP Echo to 172.16.12.1, timeout 5000 msec, TTL 64
Type Control-c to abort
Reply from 172.16.12.1 : bytes=16 time=1ms TTL=64
Reply from 172.16.12.1 : bytes=16 time=3ms TTL=64
Reply from 172.16.12.1 : bytes=16 time=1ms TTL=64
Reply from 172.16.12.1 : bytes=16 time=9ms TTL=64
Reply from 172.16.12.1 : bytes=16 time=1ms TTL=64
Reply from 172.16.12.1 : bytes=16 time=3ms TTL=64
Reply from 172.16.12.1 : bytes=16 time=62ms TTL=64
Reply from 172.16.12.1 : bytes=16 time=3ms TTL=64
Reply from 172.16.12.1 : bytes=16 time=1ms TTL=64
Reply from 172.16.12.1 : bytes=16 time=1ms TTL=64
Reply from 172.16.12.1 : bytes=16 time=1ms TTL=64
Reply from 172.16.12.1 : bytes=16 time=157ms TTL=64
Reply from 172.16.12.1 : bytes=16 time=1ms TTL=64
Reply from 172.16.12.1 : bytes=16 time=1ms TTL=64
Reply from 172.16.12.1 : bytes=16 time=1ms TTL=64
Reply from 172.16.12.1 : bytes=16 time=1ms TTL=64
Only the last 16 ping replies were printed.
Success rate is 100 percent (60/60), round-trip min/avg/max=1/6/157 ms.

Re: ICX switches have an icmp response time with significant delay

sunny_gupta — Mon, 08 Feb 2021 15:03:54 GMT

Please note that RUCKUS Fastiron code which runs on ICX treats ICMP traffic directed to the switch as low priority since it can impact the CPU. The better test case would be to ping through the switch rather than to the switch in order to get a good view of latency in the network.

Please try to ping a device through the switch and let us know if that is also experiencing any high latency.

Re: ICX switches have an icmp response time with significant delay

kshal — Mon, 08 Feb 2021 15:27:03 GMT

@sandeep_gupta_89lya7d7doyxy

Yes, we tried to do that earlier, and icmp delay increased synchronously on different VLANs.
And icmp packet loss is synchronously too.

Re: ICX switches have an icmp response time with significant delay

sunny_gupta — Mon, 08 Feb 2021 15:32:16 GMT

If you are still experiencing loss/latency then I would check the links and ensure there are no errors. If fibre, clean the glass, if copper confirm the cabling is within spec and passed.

There is no reason to have a direct connection through a switch with high latency unless the end devices are the culprit. ICX switching is line-rate and wire-speed especially in a low-utilized environment.

If you check all these and still have issues, do open a support ticket with RUCKUS TAC to investigate futher/debug.

Re: ICX switches have an icmp response time with significant delay

kshal — Mon, 08 Feb 2021 18:38:46 GMT

We have two networks - CCTV and LAN, both on ICX-7750, and both have this issue.

I'm not sure that approx hundred uplinks are dirty, or failed.

All counters on physical interfaces are looking good (statistic on most loaded interface is attached below).

Is it any deep-dive diagnostic command to detect HW problem, packet dropping etc?

Port 1/1/31 Counters:
InOctets 72871300057 OutOctets 103956128617
InPkts 122940850 OutPkts 131590140
InBroadcastPkts 174064 OutBroadcastPkts 175578
InMulticastPkts 1482989 OutMulticastPkts 49591
InUnicastPkts 121283798 OutUnicastPkts 131364971
InBadPkts 0
InFragments 0
InDiscards 0 OutErrors 0
CRC 0 Collisions 0
InErrors 0 LateCollisions 0
InGiantPkts 0
InShortPkts 0
InJabber 0 OutDiscards 0
InFlowCtrlPkts 0 OutFlowCtrlPkts 0
InBitsPerSec 138409264 OutBitsPerSec 196845056
InPktsPerSec 29588 OutPktsPerSec 31538
InUtilization 14.30% OutUtilization 20.17%

Re: ICX switches have an icmp response time with significant delay

sunny_gupta — Mon, 08 Feb 2021 20:42:03 GMT

So if it is systemic across many ports and devices, you may have some other problem. Its definitely best to work with TAC Support - you could possibly be running in to an issue that might be resolved with a bug fix/software update.

Sorry i see you are running 8.0.95c - which is very new. You may be better off going to target path unless there are features you need. i believe it would be 8.0.90x but would have to validate which is what you moved from.. so its a bit of a strange issue.

Re: ICX switches have an icmp response time with significant delay

kshal — Tue, 09 Feb 2021 15:29:27 GMT

It was done to go from "Draft version of Global RSTP (IEEE 802.1W/D3)" to RSTP.

Re: ICX switches have an icmp response time with significant delay

matt_klouda — Fri, 12 Feb 2021 19:14:02 GMT

@kshal - I just read over this thread a few times to make 100% sure I didn't miss the context, but is this comment meaning you' are running rstp on your spanning-tree links?

There shouldn't be any issues sharing this information here, but can you also provide a show default values?

Re: ICX switches have an icmp response time with significant delay

kshal — Fri, 12 Feb 2021 19:35:04 GMT

Hi Matt, see below:

show default values 
sys log buffers:4000       mac age time:300 sec       telnet sessions:5        

ip arp age:10 min          bootp relay max hops:4     ip ttl:64 hops           
ip addr per intf:24      

when multicast enabled : 
igmp group memb.:260 sec   igmp query:125 sec         hardware drop: enabled   

when ospf enabled :      
ospf dead:40 sec           ospf hello:10 sec          ospf retrans:5 sec       
ospf transit delay:1 sec 

when bgp enabled :       
bgp local pref.:100        bgp keep alive:60 sec      bgp hold:180 sec         
bgp metric:10              bgp local as:1             bgp cluster id:0         
bgp ext. distance:20       bgp int. distance:200      bgp local distance:200   

System Parameters    Default    Maximum    Current    Configured
ip-arp               8192       64000      8192       8192      
ip-static-arp        512        1024       512        512       
ip-cache             8192       32768      8192       8192      
ip-filter-port       2048       2048       2048       2048      
ip-filter-sys        8192       8192       8192       8192      
l3-vlan              32         1024       32         32          
ip-qos-session       1024       16000      1024       1024      
mac                  32768      32768      32768      32768     
ip-route             98304      131072     98304      98304     
ip-static-route      64         2048       64         64        
vlan                 1024       4095       1024       1024      
spanning-tree        128        254        128        128       
mac-filter-port      256        256        256        256       
mac-filter-sys       3072       3072       3072       3072      
ip-subnet-port       24         128        24         24        
session-limit        65536      160000     65536      65536     
view                 10         65535      10         10        
virtual-interface    255        512        255        255       
hw-ip-next-hop       30720      30720      30720      30720     
hw-traffic-condition 50         1024       50         50        
rmon-entries         2048       32768      2048       2048      
igmp-snoop-mcache    512        6144       512        512       
mld-snoop-mcache     512        6144       512        512       
ip6-route            5120       7168       5120       5120      
ip6-static-route     64         1024       64         64        
ip6-cache            1024       2048       1024       1024      
msdp-sa-cache        4096       8192       4096       4096      
gre-tunnels          16         64         16         16        
ip-vrf               128        128        128        128         
ip-route-default-vrf 65536      131072     65536      65536     
ip6-route-default-vr 2048       7168       2048       2048      
ip-route-vrf         4096       131072     4096       4096      
ip6-route-vrf        1024       7168       1024       1024      
pim-hw-mcache        1024       6144       1024       1024      
pim6-hw-mcache       512        2048       512        512       
igmp-snoop-group-add 4096       8192       4096       4096      
mld-snoop-group-addr 4096       8192       4096       4096      
mac-notification-buf 4000       16000      4000       4000      
openflow-flow-entrie 3072       12288      3072       3072      
openflow-pvlan-entri 40         40         40         40        
openflow-unprotected 40         40         40         40        
max-ecmp             8          32         8          8         
max-ip-mac           120        248        120        120       
pms-global-pool      8192       8192       8192       8192      
ip6-neighbor         4096       4096       4096       4096

Re: ICX switches have an icmp response time with significant delay

matt_klouda — Fri, 12 Feb 2021 19:41:30 GMT

Everything looks good - with your spanning-tree configuration, per vlan are you running stp, rstp, or 802-1w?