topic Re: ICX7150-48 Fails to boot - UBIFS error and Soft Lockup in RUCKUS Support for Lennar Homes

ICX7150-48 Fails to boot - UBIFS error and Soft Lockup

neonlurch — Wed, 28 Dec 2022 22:30:22 GMT

I have a switch that is failing to boot properly. I've tried reloading the Uboot, primary and secondary partitions via TFTP with new or updated code and get the same error during the boot sequence. Followed by the CPU Softlockup messages.

Starting kernel ...

....

[BrcdSoftlockup]: BUG: soft lockup - CPU#0 stuck for 22s! [mount:604]

Re: ICX7150-48 Fails to boot - UBIFS error and Soft Lockup

Yarenis — Thu, 29 Dec 2022 15:01:09 GMT

Hi @neonlurch

Please perform a software recovery, if the Switch stills not responding, open a case if you are an enterprise client.

Recovering Software

https://docs.commscope.com/bundle/fastiron-08090-upgradeguide/page/GUID-9B5D5A56-039D-44FB-8FD2-B9B55E71ADE8.html

Ruckus ICX Recovery Load boot code from boot monitor tutorial video

https://www.youtube.com/watch?v=RHriOYYD_iI&t=7s

Best regards,

Yarenis Hernández.

Technical Support Engineer | L2 TAC Wired | Lennar Home Community

Re: ICX7150-48 Fails to boot - UBIFS error and Soft Lockup

neonlurch — Thu, 29 Dec 2022 00:52:39 GMT

That is what I followed previously with a couple different versions of code. I'll give it a try again tomorrow.

As I'm not the original owner of this switch and don't have support, support wasn't able to open a case for me.

Re: ICX7150-48 Fails to boot - UBIFS error and Soft Lockup

neonlurch — Thu, 29 Dec 2022 01:03:25 GMT

I also found the following articles which may be the same issue, but can't see if they have any additional information.

https://cloudsupport.ruckuswireless.com/answers/000009411

https://cloudsupport.ruckuswireless.com/answers/000006514

Re: ICX7150-48 Fails to boot - UBIFS error and Soft Lockup

Yarenis — Thu, 29 Dec 2022 15:03:06 GMT

Hi @neonlurch

Those defects are for licensing and an old code 08061, and the Lennar home switches does not use any type of license, now take in consideration that is not just about to move the Switch from boot mode, also an upgrade must be run to the recommended code 08095g UFI, now we recommend performing the software recovery with the code 08090k, if today when perform the software recovery the Switch still no booting from any partition, please share the all the logs during the process since you did the configuration until you perform the upgrade in boot mode in both partitions.

Note: I thought you were enterprise clients, only enterprise clients can open cases.

Best regards,

Yarenis Hernández.

Technical Support Engineer | L2 TAC Wired | Lennar Home Community

Re: ICX7150-48 Fails to boot - UBIFS error and Soft Lockup

neonlurch — Thu, 29 Dec 2022 16:30:02 GMT

So this switch has that same 08061 code in the primary partition. I tried re-imaging it yesterday with that same code. Here is where I'm at today with 90k in the secondary.

Ruckus Wireless Bootloader: 10.1.15T225 (Jan 30 2019 - 23:08:47 -0800)

Booted from partition 2
DRAM: Validate Shmoo parameters stored in flash ..... OK

ICX7150-48 FULL (POE), PVT1
SYS CPLD VER: 0x4 Released Ver: 0x7

device 0 offset 0xbfe00, size 0x200
device 0 offset 0x0, size 0xc0000
Enter 'b' to stop at boot monitor: 0
ICX7150-Boot>?
? - alias for 'help'
boot - boot default, i.e., run 'bootcmd'
boot_primary- boot from primary partition

boot_secondary- boot from secondary partition

bootm - boot application image from memory
cp - memory copy
cpld_read- Read from cpld
cpld_write- write to cpld
dump_cpld- Dump CPLD Registers
eeprom - EEPROM sub-system
ext2load- load binary file from a Ext2 filesystem
ext2ls - list files in a directory (default /)
help - print command description/usage
hwinfo - Show HW Info
i2c - I2C sub-system
logging_disable- Disable Uboot Logs.
logging_enable- Enable Uboot Logs.
md - memory display
nand - NAND sub-system
ping - send ICMP ECHO_REQUEST to network host
powercycle- Do a Hard Reset
printenv- print environment variables
reset - Perform RESET of the CPU
saveenv - save environment variables to persistent storage
setenv - set environment variables
sf - SPI flash sub-system
show_image- Show Stored Images.
tftpboot- boot image via network using TFTP protocol

update_primary- update primary partition

update_secondary- update secondary partition

update_uboot- - get the uboot image over tftp.

usb - USB sub-system
verify - Verify Image
version - print monitor, compiler and linker version
ICX7150-Boot>show_image
===========================
Primary :
NAND read: device 0 offset 0x0, size 0x200
512 bytes read: OK
SPR08061c(28337116)
Secondary:
NAND read: device 0 offset 0x4000000, size 0x200
512 bytes read: OK
SPR08090k(33554432)
===========================
ICX7150-Boot>printenv
baudrate=9600
ipaddr=172.16.1.235
serverip=172.16.1.81
netmask=255.255.255.0
uboot=mnz10115.bin
image_name=SPR08090k.bin
ver=10.1.15T225 (Jan 30 2019 - 23:08:47 -0800)

Environment size: 168/16379 bytes
ICX7150-Boot>update_uboot
Loading image to Uboot Partition 1
bcm_xgs_gmac-0 Waiting for PHY auto negotiation to complete......... TIMEOUT !
Auto-neg error, defaulting to 10BT/HD
Change GMAC speed to 10MB
Using bcm_xgs_gmac-0 device
TFTP from server 172.16.1.81; our IP address is 172.16.1.235
Filename 'mnz10115.bin'.
Load address: 0x61007dc0
Loading: *
ARP Retry count exceeded; starting again
ICX7150-Boot>update_uboot
Loading image to Uboot Partition 1
Change GMAC speed to 1000MB
Using bcm_xgs_gmac-0 device
TFTP from server 172.16.1.81; our IP address is 172.16.1.235
Filename 'mnz10115.bin'.
Load address: 0x61007dc0
Loading: ######################################################
4.2 MiB/s
done
Bytes transferred = 786944 (c0200 hex)
sf erase 0x0 0xc0000
copying uboot image to flash, it will take sometime...
sf write 0x61007fc0 0x0 0xc0000
device 0 offset 0x0, size 0xc0000
TFTP to Flash Done.
ICX7150-Boot>setenv image_name=SPS08090k.bin
## Error: illegal character '='in variable name "image_name=SPS08090k.bin"
ICX7150-Boot>setenv image_name SPS08090k.bin
ICX7150-Boot>update_secondary ?
Updating SECONDARY image over TFTP
Change GMAC speed to 1000MB
Using bcm_xgs_gmac-0 device
TFTP from server 172.16.1.81; our IP address is 172.16.1.235
Filename 'SPS08090k.bin'.
Load address: 0x70000000
Loading: #################################################################
###################################################
5.3 MiB/s
done
Bytes transferred = 29360128 (1c00000 hex)
img_sz=0x1c00000 block_count = 0x1c

NAND erase.spread: device 0 offset 0x4000000, size 0x1c00000
Skipping bad block at 0x05a00000
Skipping bad block at 0x05b00000
Erasing at 0x5d00000 -- 100% complete.
OK

NAND write: device 0 offset 0x4000000, size 0x1c00000
Skip bad block 0x05a00000
Skip bad block 0x05b00000
29360128 bytes written: OK
TFTP to Flash Done.
ICX7150-Boot>boot
boot boot_primary boot_secondary bootd bootelf bootm bootp bootvx bootz
ICX7150-Boot>boot_secondary
device 0 offset 0x0, size 0xc0000
BOOTING image from Secondary

NAND read: device 0 offset 0x4000000, size 0x2000000
Skipping bad block 0x05a00000
Skipping bad block 0x05b00000
33554432 bytes read: OK
Warning: Uboot partition switched in this reload-cycle !!!
To boot Switch/Router image, reset the system...
ICX7150-Boot>powercycle

Ruckus Wireless Bootloader: 10.1.15T225 (Jan 30 2019 - 23:08:47 -0800)

Booted from partition 1
DRAM: Validate Shmoo parameters stored in flash ..... OK

ICX7150-48 FULL (POE), PVT1
SYS CPLD VER: 0x4 Released Ver: 0x7
Writing Shmoo values into flash .....

device 0 offset 0xbfe00, size 0x200
device 0 offset 0x0, size 0xc0000
Enter 'b' to stop at boot monitor: 0
device 0 offset 0x0, size 0xc0000
bootdelay: ===
Booting image from Secondary

NAND read: device 0 offset 0x4000000, size 0x2000000
Skipping bad block 0x05a00000
Skipping bad block 0x05b00000
33554432 bytes read: OK
## Loading kernel from FIT Image at 70000200 ...
Using 'conf@1' configuration
Trying 'kernel@2' kernel subimage
Description: Ruckus Linux MN VER=08.0.90k
Type: Kernel Image
Compression: lzma compressed
Data Start: 0x7024fcd0
Data Size: 2401842 Bytes = 2.3 MiB
Architecture: ARM
OS: Linux
Load Address: 0x61008000
Entry Point: 0x61008000
Hash algo: crc32
Hash value: c027c7ec
## Loading kernel from FIT Image at 70000200 ...
Using 'conf@1' configuration
Trying 'kernel@2' kernel subimage
Description: Ruckus Linux MN VER=08.0.90k
Type: Kernel Image
Compression: lzma compressed
Data Start: 0x7024fcd0
Data Size: 2401842 Bytes = 2.3 MiB
Architecture: ARM
OS: Linux
Load Address: 0x61008000
Entry Point: 0x61008000
Hash algo: crc32
Hash value: c027c7ec
Verifying Hash Integrity ... crc32+ OK
## Loading ramdisk from FIT Image at 70000200 ...
Using 'conf@1' configuration
Trying 'ramdisk@1' ramdisk subimage
Description: Ramdisk -rootfs
Type: RAMDisk Image
Compression: lzma compressed
Data Start: 0x704a0cb8
Data Size: 23843872 Bytes = 22.7 MiB
Architecture: ARM
OS: Linux
Load Address: 0x00000000
Entry Point: 0x00000000
Hash algo: crc32
Hash value: 1e3b36e9
Verifying Hash Integrity ... crc32+ OK
## Loading fdt from FIT Image at 70000200 ...
Using 'conf@1' configuration
Trying 'fdt@3' fdt subimage
Description: Flattened Device Tree blob
Type: Flat Device Tree
Compression: uncompressed
Data Start: 0x7049ea60
Data Size: 8607 Bytes = 8.4 KiB
Architecture: ARM
Hash algo: crc32
Hash value: 6fc9a37b
Verifying Hash Integrity ... crc32+ OK
Loading fdt from 0x7049ea60 to 0x65008000
Booting using the fdt blob at 0x65008000
Uncompressing Kernel Image ... OK
Using Device Tree in place at 65008000, end 6500d19e

Starting kernel ...

Mounting file system...
booting with 4.4 kernel
NAND Type: Micron NAND 2GiB (x 1)
PLATFORM MN
dd cmd: UBI
ubi0 error: ubi_io_read: error -74 (ECC error) while reading 1040384 bytes from PEB 1272:8192, read 1040384 bytes
UBIFS error (ubi0:0 pid 604): ubifs_recover_leb: corruption -3
UBIFS error (ubi0:0 pid 604): ubifs_check_node: bad node length 525360
UBIFS error (ubi0:0 pid 604): ubifs_check_node: bad node at LEB 44:786432
magic 0x6101831
crc 0xb325c4fa
node_type 1 (data node)
group_type 0 (no node group)
sqnum 36028797023622018
len 525360
key (16777579, data, 595)
size 1049600
compr_typ 0
data size 525312
data:
00000000: 3f 83 12 e7 ee 44 13 34 4d 87 53 22 94 7f bd e6 e7 c9 0f e7 f6 62 1c bf e7 20 b6 ....
[BrcdSoftlockup]: BUG: soft lockup - CPU#0 stuck for 22s! [mount:604]
00001780: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
...
[BrcdSoftlockup]: BUG: soft lockup - CPU#0 stuck for 26s! [mount:604]

...

[BrcdSoftlockup]: BUG: soft lockup - CPU#0 stuck for 663s! [mount:604]
00030500: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
00030520: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff

It continues to cycle through these same messages for as long as I let it go.

Re: ICX7150-48 Fails to boot - UBIFS error and Soft Lockup

Yarenis — Thu, 29 Dec 2022 17:00:48 GMT

Hi @neonlurch

Thank you for the details provided, based on the troubleshooting you have done since the software recovery is not working, it seems both partitions are damaged, so this unit must be RMA (return merchandise authorization ), which means to replace it, contact us via chat, I will add the link, comment that you already have a post that we have approved the RMA for you, copy the link from this post and paste it into the chat to initiate the unit replacement.

Link for the chat:

https://support.ruckuswireless.com/contact-us

Note: Please feel free to mark the post as ACCEPTED SOLUTIONS

Best regards,

Yarenis Hernández.

Technical Support Engineer | L2 TAC Wired | Lennar Home Community

Re: ICX7150-48 Fails to boot - UBIFS error and Soft Lockup

neonlurch — Thu, 29 Dec 2022 18:28:26 GMT

Okay case 01420642 has been opened.

I did let the switch continue and it somewhat booted, but as you said it looks like the flash is corrupted.

[BrcdSoftlockup]: BUG: soft lockup - CPU#0 stuck for 855s! [mount:604]
---[ end trace 6d4b128534651418 ]---
note: mount[604] exited with preempt_count 1
Segmentation fault

Mounting Config partition failed, non-recoverable file system corruption
Reformatting the flash, please download config and keys again ...

ubidetach: error!: cannot detach mtd2
error 16 (Device or resource busy)
ubiformat: error!: please, first detach mtd2 (/dev/mtd2) from ubi0
Formatting Doneubi0 error: ubi_attach_mtd_dev: mtd2 is already attached to ubi0

ubiattach: error!: cannot attach mtd2
error 17 (File exists)
ubimkvol: error!: UBI device does not have free logical erasebloubi1: attaching mtd3
cks
!!!ERROR:ubimkvol failed for config partition

Re: ICX7150-48 Fails to boot - UBIFS error and Soft Lockup

Yarenis — Thu, 29 Dec 2022 18:53:07 GMT

Hi @neonlurch

Sure, not worries the unit will be replaced, normally take one day to arrive, and you will return the damage one, take in consideration that if the unit is not running the code 08095g, please upgrade it.

Best regards,

Yarenis Hernández.

Technical Support Engineer | L2 TAC Wired | Lennar Home Community

Re: ICX7150-48 Fails to boot - UBIFS error and Soft Lockup

bilalkashif806 — Thu, 18 Dec 2025 10:02:31 GMT

i have a rukus switch i need SPR08090k.bin to recover from ghost mode.

please help