Switch Stuck in Reboot Cycle

[GrndhogsDay]

We recently had storms that affected our power and we're seeing issues with our 1 switch, 2 AP system for the first time in three years.

When the system is working it looks like this:

One access point shows four lights illuminated solid green (Power, CTL, 2.5G, 5G)

The other access point shows three lights illuminated (Power, 2.5G, 5G) with the CTL light not illuminated at all.

The switch has the STAT light illuminated green within the outline on the far left. And the SYST and Power lights are illuminated green in the array of 6 lights with no outline. The ports are blinking green randomly.

The system stays on like this for a random amount of time, sometimes 10 minutes, and sometimes 3 hours, but eventually it begins displaying different light patterns on the switch. At that point the system starts to look like this:

[Same status] One access point shows four lights illuminated solid green (Power, CTL, 2.5G, 5G)

[Same status] The other access point shows three lights illuminated (Power, 2.5G, 5G) with the CTL light not illuminated at all.

[New status] The switch has the SPD light illuminated green within the outline on the far left. The SYST light is flashing amber. The Power light is illuminated green. The ports are blinking green randomly.

This carries on until it all turns off. At that point the system looks like this:

[New status] One access point shows no lights

[New status] The other access point shows no lights

[New status] The SYST light is steady illuminated amber. The Power light is illuminated green. The ports are all off.

Then it reboots back to the first statuses I listed above when the system is "working" and the WiFi functions for another random amount of time until the cycle starts again.

After looking around the forum I believed my problem was most similar to this:

https://community.ruckuswireless.com/t5/RUCKUS-Support-for-Lennar-Homes/How-to-solve-the-routing-cod... 

I followed those instructions and was able to "see flash" on the switch console, copy the SPS one, and the final result of the switch flash says this (it's not allowing me to upload a picture so you'll need to believe that I copied this from a screenshot of the console):

ICX7150-C12 Switch#show flash

Stack unit 1:

NAND Type: Micron NAND 2GiB (x 1)

Compressed Pri Code size = 25968884, Version: 08.0.80dT211 (SPS08080d.bin)

Compressed Sec Code size = 25968884, Version: 08.0.80dT211 (SPS08080d.bin)

Compressed Boot-Monitor Image size = 786944, Version: 10.1.14T225

Code Flash Free Space = 1306181632

however on bootup of the console it seemed to be getting stuck on powering up the ports. I watched the console and it gave a "TFTP session timed out" message after saying "PoE: Power emable on port 1/1/2" which you can see in this screenshot (again, not allowing uploads of images so I'll copy the text):

ICX7150-C12 Switch>Error: Certificate is invalid

Info: Device certificate import is failed ..!!, ret: 21

Creating certificate, please wait...

SSL Client Certificate is successfully created

PS 1, Internal Power supply detected.

PS 1, Internal Power supply is up.

PoE: Stack unit 1 PS 1, Internal Power supply with 124000 mwatts capacity is up PoE Info: Adding new 54V capacity of 124000 mW, total capacity is 124000, total free capacity is 124000 PoE Info: PoE module 1 of Unit 1 on ports 1/1/1 to 1/1/12 detected. Initializing

2000 log entries of PoE Event Trace Log Buffer is allocated on unit 1 for unit 1

PoE Event Trace Logging enabled for unit 1

PoE Info: PoE module 1 of Unit 1 initialization is done.

PoE: Power enabled on port 1/1/1.

PoE: Power enabled on port 1/1/2.

TFTP session timed out

From then on it keeps repeating "TFTP session timed out" until it fails completely and reboots again. 

Please help! This is a problem that cannot be solved by anything less than a power user! 

[Imran_ruckus]

Hi @GrndhogsDay 

Greetings!!!

Thank you for the response and the detailed information.

As you are getting, please configure the Default gateway IP as described in the software recovery guide. (Please refer to the below screenshots)

And after configuring the default gateway, if you still get the ‘ping not alive’ error please turn off the firewall and repeat the software recovery process once again.

Please refer to the below link on how to turn off the Windows firewall:

https://support.microsoft.com/en-us/windows/turn-microsoft-defender-firewall-on-or-off-ec0844f7-aebd...

When you turn off the firewall, please close the putty and restart it once again, restart the software recovery process from setenv IP configuration.

Please let me know if you stuck at any point in the software recovery process.

Thank you again for your patience and understanding.

Best regards,

Imran Sanadi

RUCKUS Lennar Home Community.

[GrndhogsDay]

Imran,

I'm not currently home but I will try turning off the firewall to allow the switch to connect to the PC's IP.

As for the "Default gateway", the values I typed into that space never stayed after hitting "OK". I noticed it when I was going taking screenshots of the settings. I had already entered it when I first input the settings. Then, when reopening that IPv4 properties window, the first two lines show correctly and the third line shows empty (as seen in my screenshot). I typed the default gateway in again, hit OK, and immediately reopened, just to see that the default gateway once again did not appear. What is the remedy if the default gateway doesn't stay saved in the IPv4 settings window?

[Imran_ruckus]

Hi @GrndhogsDay 

Thank you for the response and the information.

IP Configuration should be saved after hitting OK. However, you could try disable the Firewall and repeat the software recovery steps.

Please check and let me know if you have any queries in this regard.

Thank you again for your patience and understanding.

Best regards,

Imran Sanadi

RUCKUS Lennar Home Community.

[GrndhogsDay]

Imran,

I'm going to be giving way too much detail because I am trying to make this helpful for whoever might run into this problem next.

I was able to get the Boot prompt to ping the tftp server on my computer at the 192.168.0.2 address after a lot of guesswork, trying to get the Default gateway to stay filled-in after hitting OK. The trick was to enter the IPv4 Default gateway number, hit OK on that window, but then not click anything else. At that point you'll have the Ethernet Properties window sitting open. I found that hitting OK on that Ethernet Properties window was what reset the Default gateway value to blank.

When I first connected it was in Switch mode, so I reloaded and got to boot mode by hitting 'b' while the countdown was going (you really have to watch the console for this).

^^^In this screenshot you can see me check the 'printenv' output to see the IPs matched still. Then I try to ping, and when it's successful I start the series of commands that are meant to update the software with 'setenv'  as the first command

 

^^^ And with that last screenshot you can see the 'reset' command and the beginnings of the Switch rebooting itself.

Now it's time to see if it powers on...

^^^^ by this last screenshot it's gone through all the regular power on stuff. But it also had to do some PoE download, which I haven't seen happen before. Then by the end it stops at a familiar place, the same line that was hanging before, 'PoE: Power enable on port 1/1/2.'

This didn't sit right with me, the switch has 12 ports and half are supposed to be PoE enabled, so I thought there should be more. I restarted the switch again and now I was being presented with a prompt for UN/PW. I Googled, here's the answer: UN is super and the PW is sp-admin

Once I was able to interact with the console again I lucked my way into guessing the 'show poe' command to give the status of all the ports:

^^^^ The status of all the ports is something I can't interpret on my own so please let me know if that looks correct to you. In a strange way I'm hoping it doesn't look correct, because I don't think that this is solved yet.

That's where I'm at now, and I'm not 100% satisfied with the "solution". At the current moment the WiFi is working, but it's with this set of status lights across the equipment:

Access Point 1 has (Power, CTL, 2.5G, 5G) illuminated green

Access Point 2 has (Power, 2.5G, 5G) illuminated green and (CTL) not illuminated

Switch has (STAT) illuminated green within the outline on the far left and (SYST, Power) illuminated green just to the right of that.

This is the same "working" status that came before a crash in the past, and I can't trust that it's settled yet. My hunch from the start was that the ports were not functioning correctly because the console was hanging on the port 1/1/2 line, which it is still doing. (I will trust it's settled if the support agent responds directly to the port status screenshot saying it looks fine across all 12 ports).  And the status lights on AP 2 have not improved, CTL has never illuminated since the start of this (The sad part is I'm still not even sure if it should be illuminated for the second AP, because it hasn't been addressed). This is the same combination of hardware lights, console readout, and "working" Wi-Fi status that lead to the reboot cycle to begin with. I won't look the gift-horse too closely in the mouth here, and I'll use the internet while it's working. But I'm also not holding my breath.

[Imran_ruckus · Accepted solution]

Hi @GrndhogsDay 

Thank you for the response and the detailed information.

As per the output of the inline power command and light status of the Switch seems normal.

And please note CTL light will illuminate only on ONE Access Point which acts as a master in Unleashed system.

Please let me know if you have any queries in this regard.

Thank you again for your patience and understanding.

Best regards,

Imran Sanadi

RUCKUS Lennar Home Community.