topic Re: icx7750 performance problem in ICX Switches

icx7750 performance problem

andr_boucher_5j — Fri, 12 Jul 2019 18:44:07 GMT

I have a icx7750 switch in a datacenter as routing device. Everything was running fine, but since we merge with a new organization, response time of the switch is very bad (ssh lag in cli) and we are now losing communication randomly with some servers behind. We added 4 access switch and 19 new router-interface ve (previously 2 switch and 49 interface ve).

cpu and memory dont seems to be a problem

can a system parameters was not set correctly (see below)?

thanks

SSH@R0202000-DISTR-CT-RT-01#sh default values
sys log buffers:50 mac age time:300 sec telnet sessions:5

ip arp age:10 min bootp relay max hops:4 ip ttl:64 hops
ip addr per intf:24

when multicast enabled :
igmp group memb.:260 sec igmp query:125 sec hardware drop: enabled

when ospf enabled :
ospf dead:40 sec ospf hello:10 sec ospf retrans:5 sec
ospf transit delay:1 sec

when bgp enabled :
bgp local pref.:100 bgp keep alive:60 sec bgp hold:180 sec
bgp metric:10 bgp local as:1 bgp cluster id:0
bgp ext. distance:20 bgp int. distance:200 bgp local distance:200

System Parameters Default Maximum Current Configured
ip-arp 8192 64000 8192 8192
ip-static-arp 512 1024 512 512
ip-cache 8192 32768 8192 8192
ip-filter-port 2047 2047 2047 2047
ip-filter-sys 3072 8192 8192 8192
l3-vlan 32 1024 32 32
ip-qos-session 1024 16000 1024 2048
mac 32768 32768 32768 32768
ip-route 98304 131072 98304 98304
ip-static-route 64 2048 64 64
vlan 64 4095 1024 1024
spanning-tree 128 254 128 128
mac-filter-port 32 256 32 32
mac-filter-sys 64 512 64 64
ip-subnet-port 24 128 24 24
session-limit 65536 160000 65536 65536
view 10 65535 10 10
virtual-interface 255 512 255 255
hw-ip-next-hop 17408 17408 17408 17408
hw-traffic-condition 50 1024 50 50
rmon-entries 2048 32768 2048 2048
igmp-snoop-mcache 512 6144 512 512
mld-snoop-mcache 512 6144 512 512
ip6-route 5120 7168 5120 5120
ip6-static-route 64 1024 64 64
ip6-cache 1024 2048 1024 1024
msdp-sa-cache 4096 8192 4096 4096
gre-tunnels 16 64 16 16
ip-vrf 128 128 128 128
ip-route-default-vrf 65536 131072 8000 8000
ip6-route-default-vr 2048 7168 2048 2048
ip-route-vrf 4096 131072 4096 4000
ip6-route-vrf 1024 7168 1024 1024
pim-hw-mcache 1024 6144 1024 1024
pim6-hw-mcache 512 2048 512 512
igmp-snoop-group-add 4096 8192 4096 4096
mld-snoop-group-addr 4096 8192 4096 4096
mac-notification-buf 4000 16000 4000 4000
openflow-flow-entrie 3072 3072 3072 3072
openflow-pvlan-entri 40 40 40 40
openflow-unprotected 40 40 40 40
max-ecmp 8 32 8 8
max-dhcp-snoop-entri 1024 3072 1024 1024
max-static-inspect-a 512 1024 512 512

Re: icx7750 performance problem

andrew_giancol1 — Fri, 12 Jul 2019 21:06:11 GMT

Any storms? any abnormal counters on any of your physical or pseudo interfaces? Possible bad LAG? does your issue with CLI lag exist moving from one device in your management plane to another? I'm unable to comment on your config, so just food for thought. Good Luck!

Re: icx7750 performance problem

BenBeck — Fri, 12 Jul 2019 21:12:54 GMT

Those values are just the just system max values. If you were maxing one of those out, you would likely see high cpu or error messages on console, Are there any clues in the syslogs during an issue? It's very tough to say based off the above information. Your best bet would be to open a support case and have us take a look.

Re: icx7750 performance problem

andr_boucher_5j — Tue, 16 Jul 2019 14:47:34 GMT

I tested switchover this morning and rebooted the 2 switch of the stack...same problem

Response time of the switch is still bad

pinging an inteface VE on the switch is bad, ping pinging a device in this subnet is <1ms...

Réponse de 10.16.32.1 : octets=32 temps=11 ms TTL=64
Réponse de 10.16.32.1 : octets=32 temps=189 ms TTL=64
Réponse de 10.16.32.1 : octets=32 temps=145 ms TTL=64
Réponse de 10.16.32.1 : octets=32 temps=661 ms TTL=64
Réponse de 10.16.32.1 : octets=32 temps=116 ms TTL=64
Réponse de 10.16.32.1 : octets=32 temps=33 ms TTL=64
Réponse de 10.16.32.1 : octets=32 temps=3 ms TTL=64
Réponse de 10.16.32.1 : octets=32 temps=10 ms TTL=64
Réponse de 10.16.32.1 : octets=32 temps=160 ms TTL=64
Réponse de 10.16.32.1 : octets=32 temps=188 ms TTL=64
Réponse de 10.16.32.1 : octets=32 temps=89 ms TTL=64
Réponse de 10.16.32.1 : octets=32 temps=107 ms TTL=64
Réponse de 10.16.32.1 : octets=32 temps=12 ms TTL=64
Réponse de 10.16.32.1 : octets=32 temps=106 ms TTL=64
Réponse de 10.16.32.1 : octets=32 temps=326 ms TTL=64
Réponse de 10.16.32.1 : octets=32 temps=78 ms TTL=64
Réponse de 10.16.32.1 : octets=32 temps=109 ms TTL=64
Réponse de 10.16.32.1 : octets=32 temps=114 ms TTL=64
Réponse de 10.16.32.1 : octets=32 temps=56 ms TTL=64
Réponse de 10.16.32.1 : octets=32 temps=79 ms TTL=64
Réponse de 10.16.32.1 : octets=32 temps=9 ms TTL=64
Réponse de 10.16.32.1 : octets=32 temps=123 ms TTL=64
Réponse de 10.16.32.1 : octets=32 temps=68 ms TTL=64
Réponse de 10.16.32.1 : octets=32 temps=48 ms TTL=64
Réponse de 10.16.32.1 : octets=32 temps=4 ms TTL=64
Réponse de 10.16.32.1 : octets=32 temps=217 ms TTL=64
Réponse de 10.16.32.1 : octets=32 temps=42 ms TTL=64
Réponse de 10.16.32.1 : octets=32 temps=15 ms TTL=64
Réponse de 10.16.32.1 : octets=32 temps<1ms TTL=64
Réponse de 10.16.32.1 : octets=32 temps=1 ms TTL=64
Réponse de 10.16.32.1 : octets=32 temps=1 ms TTL=64
Réponse de 10.16.32.1 : octets=32 temps=86 ms TTL=64
Réponse de 10.16.32.1 : octets=32 temps=23 ms TTL=64
Réponse de 10.16.32.1 : octets=32 temps<1ms TTL=64
Réponse de 10.16.32.1 : octets=32 temps=4 ms TTL=64
Réponse de 10.16.32.1 : octets=32 temps=12 ms TTL=64
Réponse de 10.16.32.1 : octets=32 temps=199 ms TTL=64

still investigating

Re: icx7750 performance problem

BenBeck — Tue, 16 Jul 2019 14:52:25 GMT

Any clues in your logs? (show log, syslog server, etc.). Also, is your CPU utilization jumping around at all?

Re: icx7750 performance problem

jles — Tue, 16 Jul 2019 18:20:06 GMT

Have you tried enable loop detection to see if there is a loop or something like that?

Re: icx7750 performance problem

andr_boucher_5j — Tue, 16 Jul 2019 18:50:35 GMT

no clues in the logs, cpu/memory in deep sleep, no loop by the way

SSH@R0202000-DISTR-CT-RT-01#sh cpu-utilization
1 percent busy, from 12549 sec ago
1 sec avg: 1 percent busy
5 sec avg: 1 percent busy
60 sec avg: 2 percent busy
300 sec avg: 1 percent busy

I plan to mode some int VE elsewhere and monitor the whole thing. Next step will be to open a case I think

thanks

Re: icx7750 performance problem

louis_torrez — Thu, 12 Dec 2019 16:46:41 GMT

Hello,

Any luck on this, I have very similiar issue.

Thanks.

Re: icx7750 performance problem

netdirekt — Mon, 03 Jun 2024 07:02:49 GMT

Yes, I am experiencing a similar problem. After adding and deleting a route entry, the device exhibits high CPU usage. When the ARP entry is cleared, the problem is resolved. This issue is definitely a firmware bug. We are currently discussing it with support.