ICX switches have an icmp response time with significant delay

[kshal]

Hi everybody, we have some strange behavior on Ruckus ICX switches.
If we try to ping core switch (stack of two ICX 7750) from user PC, or just from access switch stack (ICX 7250) response time can be significantly varies.
It could be 150ms, or 500ms, or some icmp packets could be without any response.
We suppose that it could be STP Topology Changes and ARP table flush, so we switched off links to another distributed switches to isolate problem.
After that we switched off secondary link from access switches to core, it doesn't help.
Finally, we leave only one active connection core-access, but icmp response time still have a jitter.
Now we have no idea how to isolate a problem.
Switch firmware was changed too, now it is 08.0.95bc (was 08.0.90dT203).
Switch utilization is 1%, uplink utilization is 1 to 3%, packets dropping are absent.
Logging buffer has no anomalies.
Network topology is a star with dual links to core from access switches stack.
Uplinks are L2 LACP.

sh cpu-utilization                
1 percent busy, from 184 sec ago
1   sec avg:  1 percent busy
5   sec avg:  1 percent busy
60  sec avg:  1 percent busy
300 sec avg:  1 percent busy

ping 172.16.12.1 count 60 interval 1
Sending 60, 16-byte ICMP Echo to 172.16.12.1, timeout 5000 msec, TTL 64
Type Control-c to abort
Reply from 172.16.12.1     : bytes=16 time=1ms TTL=64
Reply from 172.16.12.1     : bytes=16 time=3ms TTL=64
Reply from 172.16.12.1     : bytes=16 time=1ms TTL=64
Reply from 172.16.12.1     : bytes=16 time=9ms TTL=64
Reply from 172.16.12.1     : bytes=16 time=1ms TTL=64
Reply from 172.16.12.1     : bytes=16 time=3ms TTL=64
Reply from 172.16.12.1     : bytes=16 time=62ms TTL=64
Reply from 172.16.12.1     : bytes=16 time=3ms TTL=64
Reply from 172.16.12.1     : bytes=16 time=1ms TTL=64
Reply from 172.16.12.1     : bytes=16 time=1ms TTL=64
Reply from 172.16.12.1     : bytes=16 time=1ms TTL=64
Reply from 172.16.12.1     : bytes=16 time=157ms TTL=64
Reply from 172.16.12.1     : bytes=16 time=1ms TTL=64
Reply from 172.16.12.1     : bytes=16 time=1ms TTL=64
Reply from 172.16.12.1     : bytes=16 time=1ms TTL=64
Reply from 172.16.12.1     : bytes=16 time=1ms TTL=64
Only the last 16 ping replies were printed.
Success rate is 100 percent (60/60), round-trip min/avg/max=1/6/157 ms.

[sunny_gupta]

So if it is systemic across many ports and devices, you may have some other problem.  Its definitely best to work with TAC Support - you could possibly be running in to an issue that might be resolved with a bug fix/software update.

Sorry i see you are running 8.0.95c - which is very new.  You may be better off going to target path unless there are features you need.  i believe it would be 8.0.90x but would have to validate which is what you moved from..  so its a bit of a strange issue.

[kshal]

It was done to go from "Draft version of Global RSTP (IEEE 802.1W/D3)" to RSTP.

[matt_klouda]

@kshal  - I just read over this thread a few times to make 100% sure I didn't miss the context, but is this comment meaning you' are running rstp on your spanning-tree links?

There shouldn't be any issues sharing this information here, but can you also provide a show default values?

[kshal]

Hi Matt, see below:

show default values 
sys log buffers:4000       mac age time:300 sec       telnet sessions:5        

ip arp age:10 min          bootp relay max hops:4     ip ttl:64 hops           
ip addr per intf:24      

when multicast enabled : 
igmp group memb.:260 sec   igmp query:125 sec         hardware drop: enabled   

when ospf enabled :      
ospf dead:40 sec           ospf hello:10 sec          ospf retrans:5 sec       
ospf transit delay:1 sec 

when bgp enabled :       
bgp local pref.:100        bgp keep alive:60 sec      bgp hold:180 sec         
bgp metric:10              bgp local as:1             bgp cluster id:0         
bgp ext. distance:20       bgp int. distance:200      bgp local distance:200   

System Parameters    Default    Maximum    Current    Configured
ip-arp               8192       64000      8192       8192      
ip-static-arp        512        1024       512        512       
ip-cache             8192       32768      8192       8192      
ip-filter-port       2048       2048       2048       2048      
ip-filter-sys        8192       8192       8192       8192      
l3-vlan              32         1024       32         32          
ip-qos-session       1024       16000      1024       1024      
mac                  32768      32768      32768      32768     
ip-route             98304      131072     98304      98304     
ip-static-route      64         2048       64         64        
vlan                 1024       4095       1024       1024      
spanning-tree        128        254        128        128       
mac-filter-port      256        256        256        256       
mac-filter-sys       3072       3072       3072       3072      
ip-subnet-port       24         128        24         24        
session-limit        65536      160000     65536      65536     
view                 10         65535      10         10        
virtual-interface    255        512        255        255       
hw-ip-next-hop       30720      30720      30720      30720     
hw-traffic-condition 50         1024       50         50        
rmon-entries         2048       32768      2048       2048      
igmp-snoop-mcache    512        6144       512        512       
mld-snoop-mcache     512        6144       512        512       
ip6-route            5120       7168       5120       5120      
ip6-static-route     64         1024       64         64        
ip6-cache            1024       2048       1024       1024      
msdp-sa-cache        4096       8192       4096       4096      
gre-tunnels          16         64         16         16        
ip-vrf               128        128        128        128         
ip-route-default-vrf 65536      131072     65536      65536     
ip6-route-default-vr 2048       7168       2048       2048      
ip-route-vrf         4096       131072     4096       4096      
ip6-route-vrf        1024       7168       1024       1024      
pim-hw-mcache        1024       6144       1024       1024      
pim6-hw-mcache       512        2048       512        512       
igmp-snoop-group-add 4096       8192       4096       4096      
mld-snoop-group-addr 4096       8192       4096       4096      
mac-notification-buf 4000       16000      4000       4000      
openflow-flow-entrie 3072       12288      3072       3072      
openflow-pvlan-entri 40         40         40         40        
openflow-unprotected 40         40         40         40        
max-ecmp             8          32         8          8         
max-ip-mac           120        248        120        120       
pms-global-pool      8192       8192       8192       8192      
ip6-neighbor         4096       4096       4096       4096      

[matt_klouda]

Everything looks good - with your spanning-tree configuration, per vlan are you running stp, rstp, or 802-1w?