topic Re: High CPU usage - 7150-48ZP in ICX Switches

High CPU usage - 7150-48ZP

ctg24 — Wed, 21 Feb 2024 10:47:40 GMT

Hello,

We have a 7150-48ZP switch on firmware SPR09010f which is primarily used for access points. Over the past week it has been alerting 30% and 50% CPU usage every few minutes. It has been fine prior to this. Could someone advise on what the cause could be or where I can go to check for any errors? The show log command doesn't appear to show much.

Any help appreciated!

Re: High CPU usage - 7150-48ZP

Mayank — Wed, 21 Feb 2024 12:17:06 GMT

Hi ctg24 ,

Thank you for posting your query !!!

I understand that you have a 7150-48ZP with Code SPR09010f and you are facing a high cpu usage issue in every few min, And you would like to check for any errors and the cause of the issue.

There can be many cause for high cpu for example given below:

High CPU Cause #1: Depletion of Hardware Resources

High CPU Cause #2: Layer Two Loop in the Network

High CPU Cause #3: Common Services: SFLOW and SNMP

High CPU Cause #4: Features that Trigger CPU Forwarding

There are a small number of features that trigger CPU forwarding / software forwarding by design. This can vary by platform. Please refer to device-specific documentation for clarification. Features that may trigger CPU forwarding include:

1. Policy-based routing
2. Egress ACLs
3. Ip-subnet VLAN
4. Protocol VLANs
5. Private VLANs
6. Uplink Switch

For high cpu troubleshooting we use the below commands:

show cpu (3 times every one minute)
show cpu task (3 times every one minute)
de (3 times)
sh stat (3 times)
show int | inc line|util (3 times every one minute)
show memory (3 times)

show arp resource
show ip route
show ip pim resource
Show ip multicast resource
show ip traffic

You can use dm raw to see what kind of packets are hitting the cpu. Here are the steps:

Note: Please use DM raw under tac supervision as it might affect production

The Below command usage vary in different scenarios for example :

* If you're connected via SSH:
If you are connected via ssh or telnet, you also need to specify the session # which can be checked with the show who command.
- Sh who
To see which SSH session you're in
- Debug destination ssh X
To direct the logs to the SSH we're in
- Debug dest ssh x
- Dm raw mode brief
- Dm raw filter none
- Dm raw max 20
- Dm raw

But in case you are connected via console then use the below process :
* If you're connected via Console session:
- Dm raw mode brief
- Dm raw filter none
- Dm raw max 20
- Dm raw

Moving Forward the dm raw utility is a debug which will show the packets that are hitting the cpu. This will tell us if there is any specific traffic that is causing the high cpu spikes. But we always suggest our customer to use this command under tac supervision.

You can also refer the below short video for more information on DM raw command :

https://youtu.be/6a18tpkE_y4?si=IZB0H3tTCQtXnL0p

Moving Forward If this issue is not resolved , Please log a ticket with the below link so that we will help you further.

https://support.ruckuswireless.com/contact-us

I hope this information helps you

Please feel free to leave us a message if any concerns

Re: High CPU usage - 7150-48ZP

jdryan — Wed, 21 Feb 2024 14:28:29 GMT

Hi ctg24 ,

Adding to the above : the alerts seen : could you let us know if they are seen over the SZ/vSZ that the switch is connected to for management and/or monitoring.

if yes, when the same is seen, could you check the switch with the above mentioned commands.
that could help isolate the trigger point/points for the issue.

Also, on the SZ the alerts would be raised on even slight spikes, and during operation : these momentary spikes can occur based on the load, and then again, if these alerts persist over a duration within a 24 hour period. then would need to have a second look.

Secondly, if the spikes hit upwards of 80%, then the above suggested deduction would help in isolation of the cause.

Do let us know if the same helps !

Re: High CPU usage - 7150-48ZP

ctg24 — Thu, 22 Feb 2024 14:21:36 GMT

Thank you both, will give this a shot tonight out of hours and let you know how it goes!

Re: High CPU usage - 7150-48ZP

ctg24 — Tue, 27 Feb 2024 11:42:40 GMT

Got tied up so didn't get a chance last night, however have now managed to run them please see links below to output for both commands and vSZ health of the switch:

https://www.pastly.de/gqrpkpwwif

https://ibb.co/KLL97NX

Thanks again!

Re: High CPU usage - 7150-48ZP

ctg24 — Tue, 27 Feb 2024 11:52:55 GMT

Also noticed that multicast and broadcast traffic changed a fair amount following on from the spikes:

https://ibb.co/gFX73JB

https://ibb.co/RCxnpTj

Re: High CPU usage - 7150-48ZP

jdryan — Fri, 01 Mar 2024 09:20:09 GMT

Hi ctg24,

Apologies for the delay in reverting.
The CPU spikes seem to overlap with the spike in traffic that was seen.
checking on the data that was collected : the system itself is not reporting issues due to loops.

Could you check once with : show ip ssh session
Check how many sessions are active or dormant. if there are more than 2 : one for the SSH you have connected to and one for the SZ,.
then, if possible, could you try : kill ssh all : over console : and check if that reduces the CPU?

If the switch is connected to SZ, it will disconnect. and it should reform the connections.

If the same does not fix the issue : then the issue is traffic and/or network related.
I would like to ask you to open a ticket with us so that we can take a look and assist with it.

reach here to open ticket with us : https://support.ruckuswireless.com/contact-us

Regards

Re: High CPU usage - 7150-48ZP

Davidd — Fri, 26 Jul 2024 12:19:02 GMT

@Mayank wrote:
Hi ctg24 ,
Thank you for posting your query !!!
I understand that you have a 7150-48ZP with Code SPR09010f and you are facing a high cpu usage issue in every few min, And you would like to check for any errors and the cause of the issue.
There can be many cause for high cpu for example given below:
High CPU Cause #1: Depletion of Hardware Resources
High CPU Cause #2: Layer Two Loop in the Network
High CPU Cause #3: Common Services: SFLOW and SNMP
High CPU Cause #4: Features that Trigger CPU Forwarding
There are a small number of features that trigger CPU forwarding / software forwarding by design. This can vary by platform. Please refer to device-specific documentation for clarification. Features that may trigger CPU forwarding include:
1. Policy-based routing
2. Egress ACLs
3. Ip-subnet VLAN
4. Protocol VLANs
5. Private VLANs
6. Uplink Switch

For high cpu troubleshooting we use the below commands:
show cpu (3 times every one minute)
show cpu task (3 times every one minute)
de (3 times)
sh stat (3 times)
show int | inc line|util (3 times every one minute)
show memory (3 times)
show arp resource
show ip route
show ip pim resource
Show ip multicast resource
show ip traffic
You can use dm raw to see what kind of packets are hitting the cpu. Here are the steps:
Note: Please use DM raw under tac supervision as it might affect production
The Below command usage vary in different scenarios for example :
* If you're connected via SSH:
If you are connected via ssh or telnet, you also need to specify the session # which can be checked with the show who command.
- Sh who
To see which SSH session you're in
- Debug destination ssh X
To direct the logs to the SSH we're in
- Debug dest ssh x
- Dm raw mode brief
- Dm raw filter none
- Dm raw max 20
- Dm raw
But in case you are connected via console then use the below process :
* If you're connected via Console session:
- Dm raw mode brief
- Dm raw filter none
- Dm raw max 20
- Dm raw
Moving Forward the dm raw utility is a debug which will show the packets that are hitting the cpu. This will tell us if there is any specific traffic that is causing the high cpu spikes. But we always suggest our customer to use this command under tac supervision.
You can also refer the below short video for more information on DM raw command :
https://youtu.be/6a18tpkE_y4?si=IZB0H3tTCQtXnL0p

Moving Forward If this issue is not resolved , Please log a ticket with the below link so that we will help you further.
https://support.ruckuswireless.com/contact-us
I hope this information helps you
Please feel free to leave us a message if any concerns

Thanks