Switch Stuck in Reboot Cycle

[GrndhogsDay]

We recently had storms that affected our power and we're seeing issues with our 1 switch, 2 AP system for the first time in three years.

When the system is working it looks like this:

One access point shows four lights illuminated solid green (Power, CTL, 2.5G, 5G)

The other access point shows three lights illuminated (Power, 2.5G, 5G) with the CTL light not illuminated at all.

The switch has the STAT light illuminated green within the outline on the far left. And the SYST and Power lights are illuminated green in the array of 6 lights with no outline. The ports are blinking green randomly.

The system stays on like this for a random amount of time, sometimes 10 minutes, and sometimes 3 hours, but eventually it begins displaying different light patterns on the switch. At that point the system starts to look like this:

[Same status] One access point shows four lights illuminated solid green (Power, CTL, 2.5G, 5G)

[Same status] The other access point shows three lights illuminated (Power, 2.5G, 5G) with the CTL light not illuminated at all.

[New status] The switch has the SPD light illuminated green within the outline on the far left. The SYST light is flashing amber. The Power light is illuminated green. The ports are blinking green randomly.

This carries on until it all turns off. At that point the system looks like this:

[New status] One access point shows no lights

[New status] The other access point shows no lights

[New status] The SYST light is steady illuminated amber. The Power light is illuminated green. The ports are all off.

Then it reboots back to the first statuses I listed above when the system is "working" and the WiFi functions for another random amount of time until the cycle starts again.

After looking around the forum I believed my problem was most similar to this:

https://community.ruckuswireless.com/t5/RUCKUS-Support-for-Lennar-Homes/How-to-solve-the-routing-cod... 

I followed those instructions and was able to "see flash" on the switch console, copy the SPS one, and the final result of the switch flash says this (it's not allowing me to upload a picture so you'll need to believe that I copied this from a screenshot of the console):

ICX7150-C12 Switch#show flash

Stack unit 1:

NAND Type: Micron NAND 2GiB (x 1)

Compressed Pri Code size = 25968884, Version: 08.0.80dT211 (SPS08080d.bin)

Compressed Sec Code size = 25968884, Version: 08.0.80dT211 (SPS08080d.bin)

Compressed Boot-Monitor Image size = 786944, Version: 10.1.14T225

Code Flash Free Space = 1306181632

however on bootup of the console it seemed to be getting stuck on powering up the ports. I watched the console and it gave a "TFTP session timed out" message after saying "PoE: Power emable on port 1/1/2" which you can see in this screenshot (again, not allowing uploads of images so I'll copy the text):

ICX7150-C12 Switch>Error: Certificate is invalid

Info: Device certificate import is failed ..!!, ret: 21

Creating certificate, please wait...

SSL Client Certificate is successfully created

PS 1, Internal Power supply detected.

PS 1, Internal Power supply is up.

PoE: Stack unit 1 PS 1, Internal Power supply with 124000 mwatts capacity is up PoE Info: Adding new 54V capacity of 124000 mW, total capacity is 124000, total free capacity is 124000 PoE Info: PoE module 1 of Unit 1 on ports 1/1/1 to 1/1/12 detected. Initializing

2000 log entries of PoE Event Trace Log Buffer is allocated on unit 1 for unit 1

PoE Event Trace Logging enabled for unit 1

PoE Info: PoE module 1 of Unit 1 initialization is done.

PoE: Power enabled on port 1/1/1.

PoE: Power enabled on port 1/1/2.

TFTP session timed out

From then on it keeps repeating "TFTP session timed out" until it fails completely and reboots again. 

Please help! This is a problem that cannot be solved by anything less than a power user! 

[Imran_ruckus]

Hi @GrndhogsDay 

Greetings!!!

Thank you for contacting RUCKUS Lennar Home Community.

Hope you are doing well!!!

Thank you for the information you shared.

As per the case description, I could understand that your RUCKUS setup is not working after a power outage.

TFTP Timeout is normal behavior after it reboots.

Could you please let us know the answers to the below queries to assist you better?

1. On your switch while it's plugged into the outlet, do you see any lights on? (Amber or green) Please let us know which lights are on. (Please refer to the below picture and share a picture of the same, showing the light status and the FULL Switch panel cable connections to the Switch ports)

2. Is your Internet Connection from the Internet Service Provider (ISP) stable and working? (You could check this by connecting to your local router Wi-Fi or from an ethernet cable from Router to a laptop/PC).

3. May I know how many Access Points are connected to the Switch? Could you please let me know the light status on the Access Points? Any of these lights are on (Please refer to the below picture and share a picture of the same)

Please help us with pictures of your ICX-7150-C12P Switch showing the status of the lights and cable connections to the Switch ports. Also, light the status of Access Points.

Before uploading the photos please log in to the RUCKUS account and you can upload the image.

Please let us know the answers to ALL the above queries or let us know if you have any queries in this regard.

Thank you again for your patience.

Best regards,

Imran Sanadi

RUCKUS Lennar Home Community.

[GrndhogsDay]

Not to be rude, but I don't believe you read my post. I describe the exact light configuration at each stage of the reboot cycle, and gave exact console readouts from my attempt to solve this myself. I can take photos of these light configurations, but it will be duplicate information from what's written above, and will take another half-hour of waiting for a reboot cycle. Please reread my post and let me know if the information is insufficient.

[Imran_ruckus]

Hi @GrndhogsDay 

Thank you for the response.

We understand your concern.

However, we need to confirm all the light status and connections with the pictures to assist you with the proper troubleshooting steps. 

Please help us with the pictures and other queries mentioned in the last post.

Thank you again for your patience and understanding.

Best regards,

Imran Sanadi

RUCKUS Lennar Home Community.

[Imran_ruckus]

Hi @GrndhogsDay 

Thank you for the detailed information you shared.

As you mentioned there are lights on PWR and SYST, if there is an amber/orange light on the "SYST" and a green light on the "PWR" on the Switch (as it is in the below picture). Meaning the Switch is stuck in the boot mode.

Basically, due to the power outage/ power spike, the switch went to a state that we call ‘boot mode’ that is not able to boot up properly. 

Please follow the below steps to fix this issue:

First, we will try to reset the Switch as described below:

1. Please try to disconnect from the Power outlet the ICX-7150 and reconnect.

2. Please try to press the reset button using a paper click or a small stick 

• Remove power from the switch.
• Press and hold the Reset button while applying power to the switch.
• After all the system LEDs flash amber, release the Reset button.

Below is the Guide and Videos link for your reference:

https://docs.commscope.com/bundle/icx7150-installguide/page/GUID-D3B5DB74-53A7-4F8D-82F9-6EB600879EE...

Video link: https://www.youtube.com/watch?v=6jHMTdo_hK4

IF RESET IS NOT WORKING, PLEASE TRY TO RESET YOUR SWITCH MULTIPLE TIMES (Like 10-15 times) AS DESCRIBED ABOVE.

IF RESET DOESN’T WORK: Please follow the below steps.

If the reset does not make any difference, we need to perform a software recovery since it can be that the image flash is corrupted in basic words the unit is not able to boot up properly and need re-install the image to boot up properly.

Below is the Guide and Video Link to perform the Software recovery process:

Please refer to the below guide and Video link: 

https://community.ruckuswireless.com/t5/RUCKUS-Support-for-Lennar-Homes/How-to-perform-a-Software-re...

Video Link: https://www.youtube.com/watch?v=Ql22blH_CGo

Please let me know if you have any queries in this regard.

Thank you again for your patience.

Best regards,

Imran Sanadi

RUCKUS Lennar Home Community.