High CPU usage - 7150-48ZP

[ctg24]

Hello,

We have a 7150-48ZP switch on firmware SPR09010f which is primarily used for access points. Over the past week it has been alerting 30% and 50% CPU usage every few minutes. It has been fine prior to this. Could someone advise on what the cause could be or where I can go to check for any errors? The show log command doesn't appear to show much.

Any help appreciated!

[Mayank]

Hi ctg24 ,

Thank you for posting your query !!!

I understand that you have a 7150-48ZP with Code SPR09010f and you are facing a high cpu usage issue in every few min, And you would like to check for any errors and the cause of the issue.

There can be many cause for high cpu for example given below:

High CPU Cause #1: Depletion of Hardware Resources

High CPU Cause #2: Layer Two Loop in the Network

High CPU Cause #3: Common Services: SFLOW and SNMP

High CPU Cause #4: Features that Trigger CPU Forwarding

There are a small number of features that trigger CPU forwarding / software forwarding by design. This can vary by platform. Please refer to device-specific documentation for clarification. Features that may trigger CPU forwarding include:

1. Policy-based routing
2. Egress ACLs
3. Ip-subnet VLAN
4. Protocol VLANs
5. Private VLANs
6. Uplink Switch

For high cpu troubleshooting we use the below commands:

show cpu (3 times every one minute)
show cpu task (3 times every one minute)
de (3 times)
sh stat (3 times)
show int | inc line|util (3 times every one minute)
show memory (3 times)

show arp resource
show ip route
show ip pim resource
Show ip multicast resource
show ip traffic

You can use dm raw to see what kind of packets are hitting the cpu. Here are the steps:

Note: Please use DM raw under tac supervision as it might affect production

The Below command usage vary in different scenarios for example :

* If you're connected via SSH:
If you are connected via ssh or telnet, you also need to specify the session # which can be checked with the show who command.
- Sh who
To see which SSH session you're in
- Debug destination ssh X
To direct the logs to the SSH we're in
- Debug dest ssh x
- Dm raw mode brief
- Dm raw filter none
- Dm raw max 20
- Dm raw

But in case you are connected via console then use the below process :
* If you're connected via Console session:
- Dm raw mode brief
- Dm raw filter none
- Dm raw max 20
- Dm raw

Moving Forward the dm raw utility is a debug which will show the packets that are hitting the cpu. This will tell us if there is any specific traffic that is causing the high cpu spikes. But we always suggest our customer to use this command under tac supervision.

You can also refer the below short video for more information on DM raw command :

https://youtu.be/6a18tpkE_y4?si=IZB0H3tTCQtXnL0p

Moving Forward If this issue is not resolved , Please log a ticket with the below link so that we will help you further.

https://support.ruckuswireless.com/contact-us

I hope this information helps you

Please feel free to leave us a message if any concerns

[jdryan]

Hi ctg24 ,

Adding to the above : the alerts seen : could you let us know if they are seen over the SZ/vSZ that the switch is connected to for management and/or monitoring. 

if yes, when the same is seen, could you check the switch with the above mentioned commands. 
that could help isolate the trigger point/points for the issue. 

Also, on the SZ the alerts would be raised on even slight spikes, and during operation : these momentary spikes can occur based on the load, and then again, if these alerts persist over a duration within a 24 hour period. then would need to have a second look. 

Secondly, if the spikes hit upwards  of 80%, then the above suggested deduction would help in isolation of the cause. 

Do let us know if the same helps ! 

[ctg24]

Thank you both, will give this a shot tonight out of hours and let you know how it goes!

[ctg24]

Got tied up so didn't get a chance last night, however have now managed to run them please see links below to output for both commands and vSZ health of the switch:

https://www.pastly.de/gqrpkpwwif

https://ibb.co/KLL97NX

Thanks again!