topic Re: Switch Stuck in Reboot Cycle in RUCKUS Support for Lennar Homes

Switch Stuck in Reboot Cycle

GrndhogsDay — Wed, 19 Jul 2023 22:30:02 GMT

We recently had storms that affected our power and we're seeing issues with our 1 switch, 2 AP system for the first time in three years.

When the system is working it looks like this:

One access point shows four lights illuminated solid green (Power, CTL, 2.5G, 5G)

The other access point shows three lights illuminated (Power, 2.5G, 5G) with the CTL light not illuminated at all.

The switch has the STAT light illuminated green within the outline on the far left. And the SYST and Power lights are illuminated green in the array of 6 lights with no outline. The ports are blinking green randomly.

The system stays on like this for a random amount of time, sometimes 10 minutes, and sometimes 3 hours, but eventually it begins displaying different light patterns on the switch. At that point the system starts to look like this:

[Same status] One access point shows four lights illuminated solid green (Power, CTL, 2.5G, 5G)

[Same status] The other access point shows three lights illuminated (Power, 2.5G, 5G) with the CTL light not illuminated at all.

[New status] The switch has the SPD light illuminated green within the outline on the far left. The SYST light is flashing amber. The Power light is illuminated green. The ports are blinking green randomly.

This carries on until it all turns off. At that point the system looks like this:

[New status] One access point shows no lights

[New status] The other access point shows no lights

[New status] The SYST light is steady illuminated amber. The Power light is illuminated green. The ports are all off.

Then it reboots back to the first statuses I listed above when the system is "working" and the WiFi functions for another random amount of time until the cycle starts again.

After looking around the forum I believed my problem was most similar to this:

https://community.ruckuswireless.com/t5/RUCKUS-Support-for-Lennar-Homes/How-to-solve-the-routing-code-issue-in-your-ICX7150-Switch/td-p/47335

I followed those instructions and was able to "see flash" on the switch console, copy the SPS one, and the final result of the switch flash says this (it's not allowing me to upload a picture so you'll need to believe that I copied this from a screenshot of the console):

ICX7150-C12 Switch#show flash

Stack unit 1:

NAND Type: Micron NAND 2GiB (x 1)

Compressed Pri Code size = 25968884, Version: 08.0.80dT211 (SPS08080d.bin)

Compressed Sec Code size = 25968884, Version: 08.0.80dT211 (SPS08080d.bin)

Compressed Boot-Monitor Image size = 786944, Version: 10.1.14T225

Code Flash Free Space = 1306181632

however on bootup of the console it seemed to be getting stuck on powering up the ports. I watched the console and it gave a "TFTP session timed out" message after saying "PoE: Power emable on port 1/1/2" which you can see in this screenshot (again, not allowing uploads of images so I'll copy the text):

ICX7150-C12 Switch>Error: Certificate is invalid

Info: Device certificate import is failed ..!!, ret: 21

Creating certificate, please wait...

SSL Client Certificate is successfully created

PS 1, Internal Power supply detected.

PS 1, Internal Power supply is up.

PoE: Stack unit 1 PS 1, Internal Power supply with 124000 mwatts capacity is up PoE Info: Adding new 54V capacity of 124000 mW, total capacity is 124000, total free capacity is 124000 PoE Info: PoE module 1 of Unit 1 on ports 1/1/1 to 1/1/12 detected. Initializing

2000 log entries of PoE Event Trace Log Buffer is allocated on unit 1 for unit 1

PoE Event Trace Logging enabled for unit 1

PoE Info: PoE module 1 of Unit 1 initialization is done.

PoE: Power enabled on port 1/1/1.

PoE: Power enabled on port 1/1/2.

TFTP session timed out

From then on it keeps repeating "TFTP session timed out" until it fails completely and reboots again.

Please help! This is a problem that cannot be solved by anything less than a power user!

Re: Switch Stuck in Reboot Cycle

Imran_ruckus — Wed, 19 Jul 2023 22:48:27 GMT

Hi @GrndhogsDay

Greetings!!!

Thank you for contacting RUCKUS Lennar Home Community.

Hope you are doing well!!!

Thank you for the information you shared.

As per the case description, I could understand that your RUCKUS setup is not working after a power outage.

TFTP Timeout is normal behavior after it reboots.

Could you please let us know the answers to the below queries to assist you better?

1. On your switch while it's plugged into the outlet, do you see any lights on? (Amber or green) Please let us know which lights are on. (Please refer to the below picture and share a picture of the same, showing the light status and the FULL Switch panel cable connections to the Switch ports)

2. Is your Internet Connection from the Internet Service Provider (ISP) stable and working? (You could check this by connecting to your local router Wi-Fi or from an ethernet cable from Router to a laptop/PC).

3. May I know how many Access Points are connected to the Switch? Could you please let me know the light status on the Access Points? Any of these lights are on (Please refer to the below picture and share a picture of the same)

Please help us with pictures of your ICX-7150-C12P Switch showing the status of the lights and cable connections to the Switch ports. Also, light the status of Access Points.

Before uploading the photos please log in to the RUCKUS account and you can upload the image.

Please let us know the answers to ALL the above queries or let us know if you have any queries in this regard.

Thank you again for your patience.

Best regards,

Imran Sanadi

RUCKUS Lennar Home Community.

Re: Switch Stuck in Reboot Cycle

GrndhogsDay — Wed, 19 Jul 2023 22:54:04 GMT

Not to be rude, but I don't believe you read my post. I describe the exact light configuration at each stage of the reboot cycle, and gave exact console readouts from my attempt to solve this myself. I can take photos of these light configurations, but it will be duplicate information from what's written above, and will take another half-hour of waiting for a reboot cycle. Please reread my post and let me know if the information is insufficient.

Re: Switch Stuck in Reboot Cycle

Imran_ruckus — Wed, 19 Jul 2023 23:00:32 GMT

Hi @GrndhogsDay

Thank you for the response.

We understand your concern.

However, we need to confirm all the light status and connections with the pictures to assist you with the proper troubleshooting steps.

Please help us with the pictures and other queries mentioned in the last post.

Thank you again for your patience and understanding.

Best regards,

Imran Sanadi

RUCKUS Lennar Home Community.

Re: Switch Stuck in Reboot Cycle

Imran_ruckus — Wed, 19 Jul 2023 23:52:20 GMT

Hi @GrndhogsDay

Thank you for the detailed information you shared.

As you mentioned there are lights on PWR and SYST, if there is an amber/orange light on the "SYST" and a green light on the "PWR" on the Switch (as it is in the below picture). Meaning the Switch is stuck in the boot mode.

Basically, due to the power outage/ power spike, the switch went to a state that we call ‘boot mode’ that is not able to boot up properly.

Please follow the below steps to fix this issue:

First, we will try to reset the Switch as described below:

1. Please try to disconnect from the Power outlet the ICX-7150 and reconnect.

2. Please try to press the reset button using a paper click or a small stick

Remove power from the switch.
Press and hold the Reset button while applying power to the switch.
After all the system LEDs flash amber, release the Reset button.

Below is the Guide and Videos link for your reference:

https://docs.commscope.com/bundle/icx7150-installguide/page/GUID-D3B5DB74-53A7-4F8D-82F9-6EB600879EE7.html

Video link: https://www.youtube.com/watch?v=6jHMTdo_hK4

IF RESET IS NOT WORKING, PLEASE TRY TO RESET YOUR SWITCH MULTIPLE TIMES (Like 10-15 times) AS DESCRIBED ABOVE.

IF RESET DOESN’T WORK: Please follow the below steps.

If the reset does not make any difference, we need to perform a software recovery since it can be that the image flash is corrupted in basic words the unit is not able to boot up properly and need re-install the image to boot up properly.

Below is the Guide and Video Link to perform the Software recovery process:

Please refer to the below guide and Video link:

https://community.ruckuswireless.com/t5/RUCKUS-Support-for-Lennar-Homes/How-to-perform-a-Software-recovery-on-an-ICX7150-switch/td-p/47483

Video Link: https://www.youtube.com/watch?v=Ql22blH_CGo

Please let me know if you have any queries in this regard.

Thank you again for your patience.

Best regards,

Imran Sanadi

RUCKUS Lennar Home Community.

Re: Switch Stuck in Reboot Cycle

GrndhogsDay — Thu, 20 Jul 2023 01:01:40 GMT

I just spent the last stretch of time taking the pictures you requested, but I'm going to skip uploading the photos of the APs and switch's status lights since you have responded to the initial post now.

I found that software recovery link through other forum posts and have tried to apply it. The result was identical to this post:https://community.ruckuswireless.com/t5/RUCKUS-Support-for-Lennar-Homes/Power-outage-SYST-light-amber-unresponsive-ICX-7150-C12P/m-p/53486/page/2

Here everything I did, with pictures. I connected to the switch by Ethernet (via a USB-C dongle) and by the switch's USB-C. Here's a photo of those connections:

And then I set up the software. I downloaded Putty and tftp64, and here are the settings:

No matter what I did the third entry space for "Default gateway" never remained populated after hitting OK, then reopening the IPv4 settings.

I must've done at least that much right because PuTTY was able to connect to the switch console. And after it did another reboot cycle I caught the prompt to enter boot mode. Here are all the screenshots of my time in the console:

4321

You can see I struggled in the beginning while it loaded into the Switch mode, but once I got it in boot mode I was able to set all of the environment variable to the IP addresses I had set elsewhere. However, the switch was never able to make a connection, giving me the "is not alive" message over and over (I did type the wrong IP a couple times, but caught it and used the correct IP for the rest of the time). I tried restarting tftp64 and triple checked my numbers, you can see my settings are correct.

I've done the factory reset through the unplug, reset button, replug you suggested, but on each reboot it continues to leave one AP with the CTL light not illuminated, and it enters another reboot cycle after long enough time.

If I could redirect back to my first post: What do you make the switch continued always stalling out with the "TFTP session timed out" message after it tried giving PoE between port 1/1/2 and 1/1/3. I can do pictures now so here's the screenshot, look towards the middle:

Re: Switch Stuck in Reboot Cycle

WiiOutHere — Thu, 20 Jul 2023 04:46:36 GMT

I’m seeing these exact same issues with the same hardware setup and this didn’t start until yesterday. All of my Ethernet connections are going down. Are you by chance in the Thornton Lennar area? I’m seeing a lot of similar posts and was wondering if there were any similarities

Re: Switch Stuck in Reboot Cycle

GrndhogsDay — Thu, 20 Jul 2023 12:04:43 GMT

I am not in a town or community called Thornton. But Lennar seems to use all the same hardware for all of these installs, with the 7150 switch and (I believe) the 510 access points. All the photos on the Lennar-specific forum have the same hardware pictures.

Re: Switch Stuck in Reboot Cycle

WiiOutHere — Thu, 20 Jul 2023 14:44:27 GMT

I see - interesting to see all of these posts occurring at the same time. Any resolution to your issue?

I’m close to replacing the entire system all together. It is absurd to me they expect customers to telnet or ssh in.

Re: Switch Stuck in Reboot Cycle

Imran_ruckus — Thu, 20 Jul 2023 15:52:10 GMT

Hi @GrndhogsDay

Greetings!!!

Thank you for the response and the detailed information.

As you are getting, please configure the Default gateway IP as described in the software recovery guide. (Please refer to the below screenshots)

And after configuring the default gateway, if you still get the ‘ping not alive’ error please turn off the firewall and repeat the software recovery process once again.

Please refer to the below link on how to turn off the Windows firewall:

https://support.microsoft.com/en-us/windows/turn-microsoft-defender-firewall-on-or-off-ec0844f7-aebd-0583-67fe-601ecf5d774f

When you turn off the firewall, please close the putty and restart it once again, restart the software recovery process from setenv IP configuration.

Please let me know if you stuck at any point in the software recovery process.

Thank you again for your patience and understanding.

Best regards,

Imran Sanadi

RUCKUS Lennar Home Community.

Re: Switch Stuck in Reboot Cycle

Imran_ruckus — Thu, 20 Jul 2023 15:53:40 GMT

Hi @WiiOutHere

Greetings!!!

Thank you for contacting RUCKUS Lennar Home Community.

Hope you are doing well

As this case is raised by @GrndhogsDay

To avoid confusion with posts, you can create your own post with the problem you're facing and a detailed description of the network issue and what you've done so that we can guide you properly.

Thank you again for understanding.

Best regards,

Imran Sanadi

RUCKUS Lennar Home Community.

Re: Switch Stuck in Reboot Cycle

GrndhogsDay — Thu, 20 Jul 2023 16:52:36 GMT

Imran,

I'm not currently home but I will try turning off the firewall to allow the switch to connect to the PC's IP.

As for the "Default gateway", the values I typed into that space never stayed after hitting "OK". I noticed it when I was going taking screenshots of the settings. I had already entered it when I first input the settings. Then, when reopening that IPv4 properties window, the first two lines show correctly and the third line shows empty (as seen in my screenshot). I typed the default gateway in again, hit OK, and immediately reopened, just to see that the default gateway once again did not appear. What is the remedy if the default gateway doesn't stay saved in the IPv4 settings window?

Re: Switch Stuck in Reboot Cycle

Imran_ruckus — Thu, 20 Jul 2023 17:07:54 GMT

Hi @GrndhogsDay

Thank you for the response and the information.

IP Configuration should be saved after hitting OK. However, you could try disable the Firewall and repeat the software recovery steps.

Please check and let me know if you have any queries in this regard.

Thank you again for your patience and understanding.

Best regards,

Imran Sanadi

RUCKUS Lennar Home Community.

Re: Switch Stuck in Reboot Cycle

GrndhogsDay — Fri, 21 Jul 2023 00:15:48 GMT

Imran,

I'm going to be giving way too much detail because I am trying to make this helpful for whoever might run into this problem next.

I was able to get the Boot prompt to ping the tftp server on my computer at the 192.168.0.2 address after a lot of guesswork, trying to get the Default gateway to stay filled-in after hitting OK. The trick was to enter the IPv4 Default gateway number, hit OK on that window, but then not click anything else. At that point you'll have the Ethernet Properties window sitting open. I found that hitting OK on that Ethernet Properties window was what reset the Default gateway value to blank.

When I first connected it was in Switch mode, so I reloaded and got to boot mode by hitting 'b' while the countdown was going (you really have to watch the console for this).

^^^In this screenshot you can see me check the 'printenv' output to see the IPs matched still. Then I try to ping, and when it's successful I start the series of commands that are meant to update the software with 'setenv' as the first command

^^^ And with that last screenshot you can see the 'reset' command and the beginnings of the Switch rebooting itself.

Now it's time to see if it powers on...

^^^^ by this last screenshot it's gone through all the regular power on stuff. But it also had to do some PoE download, which I haven't seen happen before. Then by the end it stops at a familiar place, the same line that was hanging before, 'PoE: Power enable on port 1/1/2.'

This didn't sit right with me, the switch has 12 ports and half are supposed to be PoE enabled, so I thought there should be more. I restarted the switch again and now I was being presented with a prompt for UN/PW. I Googled, here's the answer: UN is super and the PW is sp-admin

Once I was able to interact with the console again I lucked my way into guessing the 'show poe' command to give the status of all the ports:

^^^^ The status of all the ports is something I can't interpret on my own so please let me know if that looks correct to you. In a strange way I'm hoping it doesn't look correct, because I don't think that this is solved yet.

That's where I'm at now, and I'm not 100% satisfied with the "solution". At the current moment the WiFi is working, but it's with this set of status lights across the equipment:

Access Point 1 has (Power, CTL, 2.5G, 5G) illuminated green

Access Point 2 has (Power, 2.5G, 5G) illuminated green and (CTL) not illuminated

Switch has (STAT) illuminated green within the outline on the far left and (SYST, Power) illuminated green just to the right of that.

This is the same "working" status that came before a crash in the past, and I can't trust that it's settled yet. My hunch from the start was that the ports were not functioning correctly because the console was hanging on the port 1/1/2 line, which it is still doing. (I will trust it's settled if the support agent responds directly to the port status screenshot saying it looks fine across all 12 ports). And the status lights on AP 2 have not improved, CTL has never illuminated since the start of this (The sad part is I'm still not even sure if it should be illuminated for the second AP, because it hasn't been addressed). This is the same combination of hardware lights, console readout, and "working" Wi-Fi status that lead to the reboot cycle to begin with. I won't look the gift-horse too closely in the mouth here, and I'll use the internet while it's working. But I'm also not holding my breath.

Re: Switch Stuck in Reboot Cycle

Imran_ruckus — Fri, 21 Jul 2023 15:43:30 GMT

Hi @GrndhogsDay

Thank you for the response and the detailed information.

As per the output of the inline power command and light status of the Switch seems normal.

And please note CTL light will illuminate only on ONE Access Point which acts as a master in Unleashed system.

Please let me know if you have any queries in this regard.

Thank you again for your patience and understanding.

Best regards,

Imran Sanadi

RUCKUS Lennar Home Community.

Re: Switch Stuck in Reboot Cycle

GrndhogsDay — Fri, 21 Jul 2023 16:56:54 GMT

Thank you for your help. I would recommend that you add disabling the firewall to the software recovery instructions. And the behavior of my "Default gateway" entry might be atypical, but it might be worth adding that as well because it wouldn't stay populated without the trick I mentioned above.

You guys have a very hard job to do with this support forum, this is sophisticated equipment that requires highly technical solutions, thank you for your responses! Lennar really over-estimates their home-buyers putting this in every house!

Re: Switch Stuck in Reboot Cycle

Imran_ruckus — Fri, 21 Jul 2023 17:08:46 GMT

Hi @GrndhogsDay

Thank you for the response and the feed back.

We will surely add the points you mentioned to the guide with all the details once we check them internally with the Team and Lab.

We appreciate your time and patience on this case.

Thank you again for contacting RUCKUS Lennar Home Community and feel free to contact us for future queries.

Have a great weekend ahead!!!

Best regards,

Imran Sanadi

RUCKUS Lennar Home Community.