topic Re: APs dropping from VSZ on firmware 6.1 in SmartZone and Virtual SmartZone

APs dropping from VSZ on firmware 6.1

Luis — Sun, 21 May 2023 23:24:32 GMT

We are using a VSZ, we noticed that when the controller reaches 100% of CPU, ALL APs drop (about 60 APs), then start reconnecting slowly (and keep dropping from time to time). We didn't have any issues on firmware version 5, and no changes in the network has been made, but once we upgraded to 6.1, this started happening. We have drastically increased the resources in the VM, they are more than enough and more than the minimum requirement.

We are really struggling with this.

Thanks

Re: APs dropping from VSZ on firmware 6.1

sanjay_kumar — Mon, 22 May 2023 05:25:56 GMT

Hi @Luis
Could you please share the below details:
vSZ-E or vSZ-H?
Number of APs connected: Online and Offline
Number of Switches connected : Online and Offline
HDD
CPU
RAM

Are there are components set to "Debug" under Monitor >> Application Logs ?

Re: APs dropping from VSZ on firmware 6.1

sanjay_kumar — Mon, 22 May 2023 05:26:36 GMT

Still I would recommend to open a support case since this is a network down situation.

Re: APs dropping from VSZ on firmware 6.1

Luis — Mon, 22 May 2023 21:46:52 GMT

Thank you Sanjay,

That's the thing, the network is not down. The access points keep working and there is no interruption in the WiFi, seems like the heartbeat is lost then they all show disconnected. they slowly reconnect, but this is not practical when we try to troubleshoot APs that are ACTUALLY down, and getting over a thousand alerts per day is also not practical.

Maybe if there is a way to increase the heartbeat wait on the APs, they might not drop if they can see the controller every 5min?

Appreciate your response.

Re: APs dropping from VSZ on firmware 6.1

Luis — Mon, 22 May 2023 21:53:42 GMT

Virtual SmartZoneHigh Scale

6.1.1.0.959

115 APs

No ruckus switches.

I found that every morning they drop at the same time 9:49am, they go online and offline throughout the day, then around 5:30pm, they stop dropping, till the next day at 9:49am.

????

Re: APs dropping from VSZ on firmware 6.1

Luis — Mon, 22 May 2023 22:50:32 GMT

Another piece of information.

Since this is a school, I've been taking notes on the more massive disconnections.

They always happen 5 min after the end of a class. Seems like a massive client roaming is triggering the CPU to go 100% and cause the APs "dropping"

Are you aware of any different behaviour in this firmware version when massive client roaming occurs? The pattern is pretty clear now.

Re: APs dropping from VSZ on firmware 6.1

sanjay_kumar — Tue, 23 May 2023 09:47:53 GMT

Hi @Luis
The Heartbeat loss event is generated after the AP is offline or when there is no communication from the for certain Hello packets.
But it does certainly look like a communication issue since the vSZ is going 100% CPU and not responding to APs.

Any update on the below information:
vSZ-E or vSZ-H?
Number of APs connected: Online and Offline
Number of Switches connected : Online and Offline
HDD
CPU
RAM
Are there are components set to "Debug" under Monitor >> Application Logs ?

Re: APs dropping from VSZ on firmware 6.1

Luis — Tue, 23 May 2023 22:01:18 GMT

Hi there Sanjay,

Virtual SmartZoneHigh Scale v6.1.1.0.959

115 APs (All online)

No ruckus switches.

HDD 150GB, 38GB used (now Thick provisioned, on version 5.2 it was Thin provisioned without issue)

CPU x4 (Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz) Shares High (Reservation 9900 MHz)

RAM 16GB (100% reserved)

Latency Sensitivity set to High (on 5.2 it was not set and we did not experience any issues)

Note: we didn't have any issues on 5.2, it all started after we migrated to 6.1

We found that after the school bell goes off every class, we get massive AP "disconnections", the only thing I see there is massive client roaming that triggers the issue. From 5pm to 9am next day, there are no disconnections, they always happen during class. Also no disconnections during weekends. The pattern there is very clear.

Thank you.

Re: APs dropping from VSZ on firmware 6.1

sanjay_kumar — Wed, 24 May 2023 01:58:14 GMT

Hi @Luis
The configuration looks fine. Plase open a support case and look at the logs and see what is causing the issue.

Re: APs dropping from VSZ on firmware 6.1

Luis — Wed, 24 May 2023 22:19:21 GMT

Thank you Sanjay,

We decided to downgrade to 5.2.2. Now the CPU only peaks at 67%. No AP drops.

We'll stay here until this is fixed in further firmware versions.

Thank you for your help.

Re: APs dropping from VSZ on firmware 6.1

sanjay_kumar — Thu, 25 May 2023 02:58:34 GMT

Hi @Luis
Sure, let us know if you face any issues.

Re: APs dropping from VSZ on firmware 6.1

Luis — Fri, 26 May 2023 01:26:30 GMT

I guess the important question is...

With the firmware at 6.1 the CPU spiked to 100% during clients roaming between class times, causing a timeout in the communication between controller and APs. Since we downgraded back to 5.2.2, the CPU spikes to 67%. Maybe it was always behaving like this but since it didn't cause any issues, it went undetected.

The question now is. At the moment we have 115 APs, but we are planning to add an extra 100. How is this going to impact the CPU? in theory our setup should handle up to 500 APs.

Do you have any reason for the spikes during massive client roaming? is this normal or is it specific to us?

The only thing stopping us from adding the next batch of 100 APs is this issue.

Any insight would be appreciated.

Re: APs dropping from VSZ on firmware 6.1

sanjay_kumar — Fri, 26 May 2023 03:14:33 GMT

Hi @Luis
Normally this issue should happen with just 115 APs and the number of clients you have. In 5.2.2 version, does the CPU always stays at 67%? Or it will be 67% (during the below scenario) and will be less during other times? If yes, then we can see the controller statistics and collect the logs to check the issue.

"They always happen 5 min after the end of a class. Seems like a massive client roaming is triggering the CPU to go 100% and cause the APs "dropping"

Re: APs dropping from VSZ on firmware 6.1

Luis — Fri, 26 May 2023 03:34:12 GMT

Hi Sanjay,

CPU at 67% only happens during class change, then it goes back down to 17%

Usually the CPU usage is low, but when most users roam at the same time, the CPU goes drastically up.

Re: APs dropping from VSZ on firmware 6.1

sanjay_kumar — Fri, 26 May 2023 04:26:12 GMT

Hi @Luis
I believe CPI spike issue still remains the same on 5.2.2 as well but it is not striking to 100% which happened in 6.1.
We can still troubleshoot this in 5.2.2, please open a support case so that we can check the setup and collect the relevant logs to check the issue.