topic Re: ICX 7150 UBIFS Error in RUCKUS Support for Lennar Homes

ICX 7150 UBIFS Error

chris_marquez_k — Thu, 21 Sep 2023 20:30:54 GMT

I've searched for this issue I am having on the board and it seems a few others have run into the same problem, but I don't know how to get support for myself so I'll reiterate quickly here.

Woke up after power outage and internet was out; no lights observed on the APs. SYST light on the switch was flashing green. I unplugged the switch and plugged it back in and no change. I then performed a hard reset as per the instructions on the forum and looked at the output from the console using PuTTY. Error is given below:

Starting kernel ...

booting with 4.4 kernel
NAND Type: Micron NAND 2GiB (x 1)
PLATFORM MN
dd cmd: UBI
ubi0 error: ubi_io_read: error -74 (ECC error) while reading 1040384 bytes from PEB 327:8192, read 1040384 bytes
UBIFS error (ubi0:0 pid 565): ubifs_recover_leb: corruption -3
UBIFS error (ubi0:0 pid 565): ubifs_check_node: bad node length 8389680
UBIFS error (ubi0:0 pid 565): ubifs_checks_node: bad node at LEB 1209:839680

There is also a soft lockup CPU#0 stuck for 34s error message as well.

I lastly followed a guide from the forum on performing a software recovery for the ICX 7150 and that did not work either.

How I can I get this thing working? All my security devices and others rely on the APs.

Re: ICX 7150 UBIFS Error

chris_marquez_k — Thu, 21 Sep 2023 20:34:30 GMT

Sorry, forgot to mention it is the C12P version, if that matters. Thank you.

Re: ICX 7150 UBIFS Error

Imran_ruckus — Thu, 21 Sep 2023 20:37:05 GMT

Hi @chris_marquez_k

Greetings!!!

Thank you for contacting RUCKUS Lennar Home Community.

Hope you are doing well!!!

Thank you for the information you shared.

As per the case description, I understand that your RUCKUS setup is not working, and Access Points do not have power after a power outage.

Could you please let us know the answers to the below queries to assist you better?

1. On your switch while it's plugged into the outlet, do you see any lights on? (Amber or green) Please let us know which lights are on. (Please refer to the below picture and share a picture of the same, showing the light status and the FULL Switch panel cable connections to the Switch ports)

2. Is your Internet Connection from the Internet Service Provider (ISP) stable and working? (You could check this by connecting to your local router Wi-Fi or from an ethernet cable from the Router to a laptop/PC).

3. May I know how many Access Points are connected to the Switch? (Please refer to the below picture and share a picture ALL of them)

Please help us with pictures of your ICX-7150-C12P Switch showing the status of the lights and cable connections to the Switch ports. Also, light the status of Access Points.

Before uploading the photos please log in to the RUCKUS account and you can upload the image.

Please let us know the answers to ALL the above queries or let us know if you have any queries in this regard.

Thank you again for your patience.

Best regards,

Imran Sanadi

RUCKUS Lennar Home Community.

Re: ICX 7150 UBIFS Error

chris_marquez_k — Thu, 21 Sep 2023 21:11:57 GMT

Hi, Imran. I am doing well hope you are too!! Happy to provide more info.

I do see lights on the ICX 7150, please see the pictures attached. The SYST light is solid amber. As I mentioned in my original post, I tried a hard reset according to the guide on this forum and it did not allow the switch to boot. I then followed another guide on this forum to do a software restore and it did not boot either. None of the 3 APs show any lights or signs of power.

I can confirm that my ISP router and modem are working properly. I was able to borrow a friends Unifi switch and connected the same cables in the same way they are connected to the ruckus switch, and the APs came to life and internet worked. I had to give the switch back, but it proves the APs work and its the ICX 7150 that is busted.AP 1 LightsAP 2 LightsAP 3 LightsICX Lights 1ICX Lights 2ICX Lights 3PuTTY Error Log

Re: ICX 7150 UBIFS Error

Imran_ruckus — Thu, 21 Sep 2023 21:26:29 GMT

Hi @chris_marquez_k

Thank you for the response and the detailed information you shared.

As you mentioned and I can see from the pictures you shared, there are lights on PWR and SYST, if there is an amber/orange light on the "SYST" and a green light on the "PWR" on the Switch (as it is in the below picture). Meaning the Switch is stuck in the boot mode.

Basically, due to the power outage/ power spike, the switch went to a state that we call ‘boot mode’ that is not able to boot up properly.

Please follow the below steps to fix this issue:

As you mentioned that you tried reset, once again we will try to reset the Switch as described below:

1. Please try to disconnect from the Power outlet the ICX-7150 and reconnect.

2. Please try to press the reset button using a paper click or a small stick

Remove power from the switch.
Press and hold the Reset button while applying power to the switch.
After all the system LEDs flash amber, release the Reset button.

Below is the Guide and Videos link for your reference:

https://docs.commscope.com/bundle/icx7150-installguide/page/GUID-D3B5DB74-53A7-4F8D-82F9-6EB600879EE7.html

Video link: https://www.youtube.com/watch?v=6jHMTdo_hK4

IF RESET IS NOT WORKING, PLEASE TRY TO RESET YOUR SWITCH MULTIPLE TIMES (Like 10-15 times) AS DESCRIBED ABOVE.

IF RESET DOESN’T WORK: Please follow the below steps.

If the reset does not make any difference, we need to perform a software recovery since it can be that the image flash is corrupted in basic words the unit is not able to boot up properly and need re-install the image to boot up properly.

Below is the Guide and Video Link to perform the Software recovery process:

Please refer to the below guide and Video link:

https://community.ruckuswireless.com/t5/RUCKUS-Support-for-Lennar-Homes/How-to-perform-a-Software-recovery-on-an-ICX7150-switch/td-p/47483

Video Link: https://www.youtube.com/watch?v=Ql22blH_CGo

Please let me know if you have any queries in this regard.

Thank you again for your patience and understanding.

Best regards,

Imran Sanadi

RUCKUS Lennar Home Community.

Re: ICX 7150 UBIFS Error

chris_marquez_k — Thu, 21 Sep 2023 22:44:25 GMT

Quick update: I tried the reboot method again 15 times. Each time I did so while connected to the console to see the status of the boot. In each of the 15 instances it threw the same error as before.

ubi0 error: ubi_io_read: error -74 (ECC error) while reading 1040384 bytes from PEB 327:8192, read 1040384 bytes
UBIFS error (ubi0:0 pid 565): ubifs_recover_leb: corruption -3
UBIFS error (ubi0:0 pid 565): ubifs_check_node: bad node length 8651824
UBIFS error (ubi0:0 pid 565): ubifs_check_node: bad node at LEB 1209:839680

I let the console run on some of the resets for several minutes and all I got was a bunch of zeros and every now and then the soft lockup error below.

[BrcdSoftlockup]: BUG: soft lockup - CPU#0 stuck for 22s! [mount:565]

I will now try the software recovery and report back. Thanks for your patience.

Re: ICX 7150 UBIFS Error

Imran_ruckus — Thu, 21 Sep 2023 23:16:35 GMT

Hi @chris_marquez_k

Thank you for the response and the update.

Yes, if resetting with a paper clip does not work, we have to perform a software recovery to the Switch as described in my previous post guide link and video.

Please note console will not work when the Switch is stuck in boot mode. Either reset with a paper clip or software recovery are the only options.

Please perform software recovery and share the screenshots if you are getting any errors or are stuck at any point in the software recovery process.

Thank you again for your patience and understanding.

Best regards,

Imran Sanadi

RUCKUS Lennar Home Community.

Re: ICX 7150 UBIFS Error

chris_marquez_k — Thu, 21 Sep 2023 23:21:40 GMT

Alright I'm running into an issue now on the software recovery attempt. I still have all the programs downloaded from this morning and the drivers. I have changed the TCP/IPv4 properties as instructed by the guide, and I launched PuTTY to change the serverip. I've connected the computer to the switch using the out of band management port. I tried the test to ping 192.168.0.2 but it says not alive. I verified the info using printenv but it still wont say 192.168.0.2 is alive. I go back into the properties of the TCP/IP and the gateway is blank even though I saved it.

ICX7150-Boot>printenv
baudrate=9600
ipaddr=192.168.0.3
serverip=192.168.0.2
netmask=255.255.255.0
gatewayip=192.168.0.2
uboot=mnz10113
image_name=SPR08070c.bin
ver=10.1.13T225 (Jul 24 2018 - 03:23:00 -0700)

Environment size: 185/16379 bytes
ICX7150-Boot>ping 192.168.0.2
Change GMAC speed to 1000MB
Using bcm_xgs_gmac-0 device
ping failed; host 192.168.0.2 is not alive

Re: ICX 7150 UBIFS Error

Imran_ruckus — Thu, 21 Sep 2023 23:24:39 GMT

Hi @chris_marquez_k

Thank you for the response and the update.

Could you please share a picture showing the software recovery cable connections between the Switch and the laptop?

Thank you again for your patience and understanding.

Best regards,

Imran Sanadi

RUCKUS Lennar Home Community.

Re: ICX 7150 UBIFS Error

chris_marquez_k — Thu, 21 Sep 2023 23:46:17 GMT

Of course. Here are the connections.

Re: ICX 7150 UBIFS Error

Imran_ruckus — Thu, 21 Sep 2023 23:50:06 GMT

Hi @chris_marquez_k

Thank you for the response and the detailed information you shared.

Connections are correct.

As there is "host 192.168.0.2 is not alive" error.

Are you turning off the Windows firewall? If not, please turn off the Windows firewall and retry the software recovery process once again.

Please refer to the below link on how to turn off the Windows firewall:

https://support.microsoft.com/en-us/windows/turn-microsoft-defender-firewall-on-or-off-ec0844f7-aebd-0583-67fe-601ecf5d774f

When you turn off the firewall, please close the putty and restart it once again, restart the software recovery process from setenv IP configuration.

Please let me know if you are getting the same error or are stuck at any point in the software recovery process.

Thank you again for your patience and understanding.

Best regards,

Imran Sanadi

RUCKUS Lennar Home Community.

Re: ICX 7150 UBIFS Error

chris_marquez_k — Fri, 22 Sep 2023 00:08:43 GMT

Thanks for the tip on the windows firewall. It was on. But that did not fix my not alive error. I did as asked; turned off firewall, restarted putty, and began the process from setting all the environment IPs. It still gave me the not alive error when I pinged 192.168.0.2

Is there somewhere else to check what the problem could be?

Re: ICX 7150 UBIFS Error

Imran_ruckus — Fri, 22 Sep 2023 00:13:54 GMT

Hi @chris_marquez_k

Thank you for the response and the information you shared.

Even after turning off the firewall if you see the same error. Please perform the next commands after this step you see in the below ‘not alive’ error.

ICX7150-Boot>ping 192.168.0.2
Change GMAC speed to 1000MB
Using bcm_xgs_gmac-0 device
ping failed; host 192.168.0.2 is not alive

Please perform the below commands:

“setenv image_name SPS08090k.bin” (enter the correct image file name)

“update_primary” (upload the image on the primary partition)

“setenv image_name SPS08090k.bin” (enter the correct image file name)

“update_secondary” (upload the image on the secondary partition)

*Wait that loads the image and continue with the boot:

“setenv uboot mnz10115.bin” (enter the correct boot file name)

“update_uboot” (upload the boot code)

* Wait that loads the boot and continue with the Final command reset:

“reset” (finally enter reset to reload the unit)

After performing the above commands please check if PWR and SYST lights on the Switch turns GREEN, meaning it is back to normal and let me know if you have any queries in this regard.

Thank you again for your patience and understanding.

Best regards,

Imran Sanadi

RUCKUS Lennar Home Community.

Re: ICX 7150 UBIFS Error

Imran_ruckus — Fri, 22 Sep 2023 00:17:38 GMT

Hi @chris_marquez_k

Please perform the commands as mentioned in my previous post and check. Share the screenshots if you are getting any errors or are stuck at any point in the software recovery process.

Thank you again for your patience and understanding.

Best regards,

Imran Sanadi

RUCKUS Lennar Home Community.

Re: ICX 7150 UBIFS Error

chris_marquez_k — Fri, 22 Sep 2023 00:30:13 GMT

Imran, I didn't get far this time. There was a timeout error as seen below after I named the file (I verified its the right file) and tried to update the primary.

Re: ICX 7150 UBIFS Error

chris_marquez_k — Fri, 22 Sep 2023 00:48:16 GMT

What else should be plugged in to the switch while attempting to change the environment to load the new software? Just want to rule out anything there that would explain why I can't seem to ping it.

Re: ICX 7150 UBIFS Error

chris_marquez_k — Fri, 22 Sep 2023 00:50:12 GMT

Sorry meant to reply to this: it timed out when I tried to update primary. But I meant to ask what else if anything should be plugged into the switch while doing this? I want to make sure everything is right.

Re: ICX 7150 UBIFS Error

chris_marquez_k — Fri, 22 Sep 2023 02:52:34 GMT

Good evening, Imran.

I was able to surpass the "not alive" hurdle by tweaking the gateway IP. (For anyone following along that is a novice like me, the guide shows the gateway IP address as being the same as the IP address when configuring the TCP/IPv4 properties. My computer was immediately clearing the gateway IP as that isn't valid.)

Anyway, once this was done, I was able to continue on with the steps of updating the primary and secondary partitions. This is the relevant code for the update_primary:
Primary Update

After updating the primary, I updated the secondary:
Secondary update

Here I noticed there are some bad blocks that were skipped.
I then updated the uboot:
Update uboot

Lastly I ran the reset command. After on the boot, it ran from the primary partition and failed all the same. Same exact error message. I unplugged the switch and booted from the secondary to ensure it was completely bricked, and it also failed to boot from the secondary.

Imran, I appreciate your assistance and walking me through this process. At this stage, I feel safe in saying the switch is in fact, corrupted. I have tried all avenues multiple times.

Can we please proceed with the RMA process to get this replaced?

Re: ICX 7150 UBIFS Error

Imran_ruckus — Fri, 22 Sep 2023 15:05:35 GMT

Hi @chris_marquez_k

Greetings!!!

Thank you for the response and the detailed information you shared.

After you perform the ‘reset’ command at the end, could you please check the status of the PWR and SYST lights on the Switch and let me know?

Please let me know if you have any queries in this regard.

Thank you again. We appreciate your patience and understanding.

Best regards,

Imran Sanadi

RUCKUS Lennar Home Community.

Re: ICX 7150 UBIFS Error

chris_marquez_k — Fri, 22 Sep 2023 16:56:45 GMT

Hi Imran good afternoon.

the PWR light is solid green and the SYST light is solid amber. It is in the exact same state as prior to the resets and software recovery attempts.