topic Re: Smartzone 100 and R730 Reboots/ Offline Issue in Wireless Questions and Best Practices

Smartzone 100 and R730 Reboots/ Offline Issue

kevin_7278725 — Wed, 09 Oct 2019 17:48:20 GMT

Is anyone having issues with R730's randomly going Offline and requiring the AP to be rebooted in order to get it back online? We have been seeing this issue for about 30 days now and support has been unable to tell us why the R730's are doing this. The AP's are inaccessible most of the time when the issue happens, but sometimes we are able to still ping the AP and actually SSH to the login prompt, but the admin account/password will fail. Once we power cycle the AP it works fine again. When the issue is happening the AP is still accepting clients but it has no network access so those clients are broken. It is very frustrating and Ruckus support has been no help. The R730's are connected to Ruckus ICX 7650's via 5Gb multigig ports. The switches report no problems and there are other AP's on the same switch at the time that have no issue, so the problem is just random Access Point specific. The issue is completely random, no pattern can be found, other than support telling us they are seeing AP kernel panics and that they can't tell us why or how to make stop.

Smartzone 100 version is 5.1.2.0.302 - which support had us upgrade to as they said that would fix the kernel panics - It has not

R730 version - 5.1.2.0.373

A few of the R730's have not been able to recover from this issue after a reboot and have had to be RMA'd. Some of them will automatically reboot after 15-30 mins, but if we manually reboot them they typically come back online and work. Was curious if any else is experiencing this issue with R730's, Smartzone 100's, and ICX 7650's?

Re: Smartzone 100 and R730 Reboots/ Offline Issue

mario_javier_ga — Wed, 09 Oct 2019 19:12:03 GMT

Hi Kevin, I have a similar situation, but the reference of AP's and controller is different, however, the behavior is the same, and like you I could not have a diagnosis by Ruckus support.

Re: Smartzone 100 and R730 Reboots/ Offline Issue

kevin_7278725 — Wed, 09 Oct 2019 19:20:25 GMT

Thanks Mario. Would you mind telling me the model AP's and version of AP and controller you are running. It may help me argue with support more as they have been very unhelpful and just keep asking for more and more logs. Which the logs are all the same and show their AP's doing kernel panics and then the behavior is either a complete lockup of the AP or a reboot. 3 times the units have bricked themselves and even though we ask for the reason on the RMA we get back that they destroyed the units and can't provide a reason as we have to ask for a reason when we open the RMA. We do that and still don't get a reason. Horrible support.

Re: Smartzone 100 and R730 Reboots/ Offline Issue

mario_javier_ga — Wed, 09 Oct 2019 19:37:56 GMT

Kevin, in the network you have AP of different references, R720, T310, among others, the difficulty arises in all references with the same behavior that you mention, we have tested with vSZ in the cloud and on-premise, and it continues With the same problem, the version currently in the group is 5.1.2.0.302, but we have experienced the same with several firmware from the first version 5.1.1.

Re: Smartzone 100 and R730 Reboots/ Offline Issue

kevin_7278725 — Wed, 09 Oct 2019 19:52:04 GMT

Do you still have your case open with support and have they provided you any guidance to fix the issue? We are getting no where with them

Re: Smartzone 100 and R730 Reboots/ Offline Issue

mario_javier_ga — Wed, 09 Oct 2019 20:02:38 GMT

I had one open for approximately 40 days, I was requested by the controller and AP's log, but the conclusion was that no fault was found, which is why we chose to migrate to an on-premise controller, but the fail, today I had to open a new case.

Re: Smartzone 100 and R730 Reboots/ Offline Issue

kevin_7278725 — Wed, 09 Oct 2019 20:12:09 GMT

Please update if they provide anything that helps as having random AP's reboot, Go Offline, or die is extremely painful.

Re: Smartzone 100 and R730 Reboots/ Offline Issue

mario_javier_ga — Wed, 09 Oct 2019 20:27:00 GMT

Of course, if Kevin, also if you find any input please update me, I have been with this problem for a long time and I have not been able to solve it. thank you

Re: Smartzone 100 and R730 Reboots/ Offline Issue

kevin_7278725 — Wed, 09 Oct 2019 20:35:49 GMT

Definitely. I am 99.9% sure the problem is a memory leak in their 5.x code which Ruckus needs to fix. I think it has to do with Wifi6 (AC) connections and something they are doing which I have not been able to pinpoint so I can intentionally cause the issue, which causes the AP's to leak memory until they crash. Why support can't figure this out and make it stop is beyond me. All they do is request more and more logs and make us run packet captures. Happy to know someone else is experiencing the issue but not happy to see how long you have been dealing without a fix.

Re: Smartzone 100 and R730 Reboots/ Offline Issue

mario_javier_ga — Wed, 09 Oct 2019 20:39:39 GMT

That's right Kevin, I hope we have positive news soon to solve the issue, best regards.

Re: Smartzone 100 and R730 Reboots/ Offline Issue

sven_kessler — Thu, 10 Oct 2019 09:59:05 GMT

We have a similar issue with R730 APs. Everything is working fine for some days but suddenly, clients do no have network access when connected to these APs. Only an AP reboot fixes the issue for some time until it happens again. We have APs on firmware 5.1.1.0.624 and vSZ-H on 5.1.1.0.598.
Right now, we replaced the APs with R510 and everything i working without issues,

Re: Smartzone 100 and R730 Reboots/ Offline Issue

kevin_7278725 — Thu, 10 Oct 2019 11:53:20 GMT

Sven, do you have a case open with support? If you don't could you please open one so they understand they are impacting multiple customers. They tried telling us we are the only customer with this issue. I am seeing now it appears to be anyone running Smartzone 5.x and AP's of 610 and higher. I don't have the luxury of replacing all our brand new R730's with anything so I need Ruckus to fix the issue. I believe the issue is related to Wifi6 (11ax) since there are more and more 11ax devices coming online in the past 30 days, especially with iPhone 11. I think Ruckus has a memory leak that they don't know how to fix and I have little faith they will fix this soon as Mario posted of the problem 4 months ago and they still have not resolved it. Their customer support for issues like this is horrible.

Re: Smartzone 100 and R730 Reboots/ Offline Issue

sven_kessler — Thu, 10 Oct 2019 12:27:49 GMT

We do not have an open case regarding this right now, because I'm afraid the "standard solution steps" will take so much time compared to the outcome, so that we currently wait for a fix hopefully within the next firmware update.
But I get your point: Is nobody opens a case, there will be no solution.
I will be on vacation next week and open a call after that.

Re: Smartzone 100 and R730 Reboots/ Offline Issue

mario_javier_ga — Thu, 10 Oct 2019 13:22:17 GMT

Hi Sven,

Thank you very much for the contribution, I think like Kevin, we hope you help us by opening a case to support from Ruckus since that will show that it is a general problem and will help to give a quick solution.

Thank you,

Re: Smartzone 100 and R730 Reboots/ Offline Issue

michael_brado — Thu, 10 Oct 2019 22:06:40 GMT

Mario, Kevin, Sven, please tell me your case numbers. We ought to be able to collect logs and AP support info to identify the cause of any problem(s), especially if you see it happening frequently.

Re: Smartzone 100 and R730 Reboots/ Offline Issue

michael_brado — Thu, 10 Oct 2019 22:21:55 GMT

Mario, I saw your ticket 985751, and it says you performed an SZ 5.1.2.0.302 upgrade. Please let us know if you see another situation, and then try to grab AP support info and SZ logs for tech support, thanks!

Re: Smartzone 100 and R730 Reboots/ Offline Issue

kevin_7278725 — Fri, 11 Oct 2019 12:08:27 GMT

Ticket 965817 - as of late yesterday our issue has finally been escalated and we are scheduled to work with an escalation engineer this afternoon who is planning on enabling some additional debugging/logging on the R730's and Smartzone. Hoping they are able to figure this out soon as we still have R730's randomly rebooting. I will provide the post updates as we make progress so others hopefully don't have to deal with this issue.

Re: Smartzone 100 and R730 Reboots/ Offline Issue

kevin_7278725 — Fri, 11 Oct 2019 13:33:33 GMT

R730, reboot, kernel panic, apHearbeatLost, smartzone - Just adding some of the keywords that I was using to search for others with this problem so that hopefully anyone else with this random access point issue might add a comment so Ruckus is aware of all the customers impacted by this issue.

Re: Smartzone 100 and R730 Reboots/ Offline Issue

ian_johnson_dfw — Fri, 11 Oct 2019 19:15:41 GMT

This is very interesting, I have 3 * R730s, two display uptime of 38days, one restarts daily. Ive been meaning to troubleshoot and this thread prompts me to do that sooner now.

Re: Smartzone 100 and R730 Reboots/ Offline Issue

kevin_7278725 — Fri, 11 Oct 2019 19:25:52 GMT

Thank Ian. Please open a case with them and tell them to reference my case 965817 for details. They are now trying to capture memory and cpu state with a custom script of the AP's prior to the reboots. Hopefully the more customers they see opening cases for the R730's rebooting they will figure out the issue and fix it. We ran R610's and R710's for 3 years without ever experiencing a random reboot. We were also running ZoneDirector instead of Smartzone so I am not sure which is the actual culprit.