Switch Stuck in Reboot Cycle

[GrndhogsDay]

We recently had storms that affected our power and we're seeing issues with our 1 switch, 2 AP system for the first time in three years.

When the system is working it looks like this:

One access point shows four lights illuminated solid green (Power, CTL, 2.5G, 5G)

The other access point shows three lights illuminated (Power, 2.5G, 5G) with the CTL light not illuminated at all.

The switch has the STAT light illuminated green within the outline on the far left. And the SYST and Power lights are illuminated green in the array of 6 lights with no outline. The ports are blinking green randomly.

The system stays on like this for a random amount of time, sometimes 10 minutes, and sometimes 3 hours, but eventually it begins displaying different light patterns on the switch. At that point the system starts to look like this:

[Same status] One access point shows four lights illuminated solid green (Power, CTL, 2.5G, 5G)

[Same status] The other access point shows three lights illuminated (Power, 2.5G, 5G) with the CTL light not illuminated at all.

[New status] The switch has the SPD light illuminated green within the outline on the far left. The SYST light is flashing amber. The Power light is illuminated green. The ports are blinking green randomly.

This carries on until it all turns off. At that point the system looks like this:

[New status] One access point shows no lights

[New status] The other access point shows no lights

[New status] The SYST light is steady illuminated amber. The Power light is illuminated green. The ports are all off.

Then it reboots back to the first statuses I listed above when the system is "working" and the WiFi functions for another random amount of time until the cycle starts again.

After looking around the forum I believed my problem was most similar to this:

https://community.ruckuswireless.com/t5/RUCKUS-Support-for-Lennar-Homes/How-to-solve-the-routing-cod... 

I followed those instructions and was able to "see flash" on the switch console, copy the SPS one, and the final result of the switch flash says this (it's not allowing me to upload a picture so you'll need to believe that I copied this from a screenshot of the console):

ICX7150-C12 Switch#show flash

Stack unit 1:

NAND Type: Micron NAND 2GiB (x 1)

Compressed Pri Code size = 25968884, Version: 08.0.80dT211 (SPS08080d.bin)

Compressed Sec Code size = 25968884, Version: 08.0.80dT211 (SPS08080d.bin)

Compressed Boot-Monitor Image size = 786944, Version: 10.1.14T225

Code Flash Free Space = 1306181632

however on bootup of the console it seemed to be getting stuck on powering up the ports. I watched the console and it gave a "TFTP session timed out" message after saying "PoE: Power emable on port 1/1/2" which you can see in this screenshot (again, not allowing uploads of images so I'll copy the text):

ICX7150-C12 Switch>Error: Certificate is invalid

Info: Device certificate import is failed ..!!, ret: 21

Creating certificate, please wait...

SSL Client Certificate is successfully created

PS 1, Internal Power supply detected.

PS 1, Internal Power supply is up.

PoE: Stack unit 1 PS 1, Internal Power supply with 124000 mwatts capacity is up PoE Info: Adding new 54V capacity of 124000 mW, total capacity is 124000, total free capacity is 124000 PoE Info: PoE module 1 of Unit 1 on ports 1/1/1 to 1/1/12 detected. Initializing

2000 log entries of PoE Event Trace Log Buffer is allocated on unit 1 for unit 1

PoE Event Trace Logging enabled for unit 1

PoE Info: PoE module 1 of Unit 1 initialization is done.

PoE: Power enabled on port 1/1/1.

PoE: Power enabled on port 1/1/2.

TFTP session timed out

From then on it keeps repeating "TFTP session timed out" until it fails completely and reboots again. 

Please help! This is a problem that cannot be solved by anything less than a power user! 

[GrndhogsDay]

I just spent the last stretch of time taking the pictures you requested, but I'm going to skip uploading the photos of the APs and switch's status lights since you have responded to the initial post now. 

I found that software recovery link through other forum posts and have tried to apply it. The result was identical to this post:https://community.ruckuswireless.com/t5/RUCKUS-Support-for-Lennar-Homes/Power-outage-SYST-light-ambe...

Here everything I did, with pictures. I connected to the switch by Ethernet (via a USB-C dongle)  and by the switch's USB-C. Here's a photo of those connections:

And then I set up the software. I downloaded Putty and tftp64, and here are the settings:

No matter what I did the third entry space for "Default gateway" never remained populated after hitting OK, then reopening the IPv4 settings.

I must've done at least that much right because PuTTY was able to connect to the switch console. And after it did another reboot cycle I caught the prompt to enter boot mode. Here are all the screenshots of my time in the console:

4321

You can see I struggled in the beginning while it loaded into the Switch mode, but once I got it in boot mode I was able to set all of the environment variable to the IP addresses I had set elsewhere. However, the switch was never able to make a connection, giving me the "is not alive" message over and over (I did type the wrong IP a couple times, but caught it and used the correct IP for the rest of the time). I tried restarting tftp64 and triple checked my numbers, you can see my settings are correct.

I've done the factory reset through the unplug, reset button, replug you suggested, but on each reboot it continues to leave one AP with the CTL light not illuminated, and it enters another reboot cycle after long enough time.

If I could redirect back to my first post: What do you make the switch continued always stalling out with the "TFTP session timed out" message after it tried giving PoE between port 1/1/2 and 1/1/3. I can do pictures now so here's the screenshot, look towards the middle:

[WiiOutHere]

I’m seeing these exact same issues with the same hardware setup and this didn’t start until yesterday. All of my Ethernet connections are going down. Are you by chance in the Thornton Lennar area? I’m seeing a lot of similar posts and was wondering if there were any similarities 

[GrndhogsDay]

I am not in a town or community called Thornton. But Lennar seems to use all the same hardware for all of these installs, with the 7150 switch and (I believe) the 510 access points. All the photos on the Lennar-specific forum have the same hardware pictures. 

[WiiOutHere]

I see - interesting to see all of these posts occurring at the same time. Any resolution to your issue?

I’m close to replacing the entire system all together. It is absurd to me they expect customers to telnet or ssh in. 

[Imran_ruckus]

Hi @WiiOutHere 

Greetings!!!

Thank you for contacting RUCKUS Lennar Home Community.

Hope you are doing well

As this case is raised by @GrndhogsDay 

To avoid confusion with posts, you can create your own post with the problem you're facing and a detailed description of the network issue and what you've done so that we can guide you properly.

Thank you again for understanding.

Best regards,

Imran Sanadi

RUCKUS Lennar Home Community.