ICX7150 fails to boot to primary partition (ICX7150-Boot and OS>)

[matt_radecic]

My ICX7150 crashed tonight and is unable to recover after flashing primary/secondary partition.

Starting kernel ...

Mounting file system...
booting with 4.4 kernel
NAND Type: Micron NAND 2GiB (x 1)
PLATFORM MN
dd cmd: UBI
Config partition mounted
Resource partition is mounted!!
eUSB Not mounted or detected
Successfully read the SKU Reg, the value is: 0xff
dma_mem_base: 0x96000000, dma_mem_len: 0x4000000, warm_mem_base: 0x9f500000, warm_mem_len: 0xb00000
Please download UFI image to install packages
/etc/process_packages.sh: line 170: can't create /.pkg/error.log: nonexistent directory
Package installation is incomplete, related functionalities may not be available
Cannot boot FastIron application due to package installation failure
starting console application
server-execvp: python3: No such file or directory

It will fail to load and boot into OS> prompt

if i type 'exit' it then hangs due to installation failure

abduco: /tmp/con_sh: session terminated with exit status 0
ICX prereq settings ..
cat: can't open '/usr/local/etc/sshd_config': No such file or directory
Setting up hostname
Setting up ISC DHCP
cp: can't stat '/.pkg/dhcpdPkg/script/dhclient-exit-hooks': No such file or directory
cp: can't stat '/.pkg/dhcpdPkg/script/dhclient-script': No such file or directory
cp: can't stat '/.pkg/dhcpdPkg/default-dhclientv4.conf': No such file or directory
ICX prereq settings done
dhcpd executable file is not present.
Starting up system statistics daemon...
Starting systatsd done
Starting up poe daemon...
Successfully mapped 4194304 space
Number of hash elements 8000
Number of key objects 4000
Number of contexts 4000
Start of Data region b6358000
Systatsd Logging Env is not set, initializing log level to: D
Setting connection to systemlogger
poed Logging Env is not set, initializing log level to: I
Setting connection to systemlogger
Starting poed done
dhcpdv4 executable file is not present.
dhclientv4 executable file is not present.
Starting up health monitor daemon...
Starting hmond done
Hmond Logging Env is not set, initializing log level to: I
Setting connection to systemlogger
cp: can't stat '/.pkg/FileMgrPkg/lib/*.so': No such file or directory
nohup: appending output to nohup.out
Backup is fine
Kernel crashdump is disabled.
ln: error while loading shared libraries: libpam.so.0: cannot open shared object file: No such file or directory
cat: error whileexec of "/.pkg/sshConfPkg/bin/ssh-keygen" failed: No such file or directory
cat: error while loading shared libraries: libpam.so.0: cannot open shared object file: No such file or directory
Setting up ecdsa key
sh: error while loading shared libraries: libpam.so.0: cannot open shared object file: No such file or directory
exec of "/.pkg/sshConfPkg/bin/ssh-keygen" failed: No such file or directory
cat: error while loading shared libraries: libpam.so.0: cannot open shared object file: No such file or directory
Starting mgmt monitor scrit
bin/sh: error while loading shared libraries: libpam.so.0: cannot open shared object file: No such file or directory
cat: error while loading shared libraries: libpam.so.0: cannot open shared object file: No such file or directory
cat: error while loading shared libraries: libpam./bin/shsh: error while loading shared libraries: libpam.so.0: cannot open shared object file: No such file or directory
sh: error while loading shared libraries: libpam.so.0: cannot open shared object file: No such file or directory
/bin/sh/bin/sh

===============================

ICX7150-Boot>printenv
baudrate=9600
ipaddr=192.168.1.1
serverip=192.168.1.251
netmask=255.255.255.0
uboot=mnz10126.bin
image_name=SPS09010jufi.bin
ver=10.1.26T225 (Nov 29 2022 - 04:43:18 -0800)

ICX7150-Boot>show_image
===========================
Primary :
NAND read: device 0 offset 0x0, size 0x200
512 bytes read: OK
SPS09010j(29360128)
Secondary:
NAND read: device 0 offset 0x4000000, size 0x200
512 bytes read: OK
SPS09010j(29360128)
===========================

ICX7150-Boot>update_primary
Updating PRIMARY image over TFTP

Load addr=0x70000000
Change GMAC speed to 1000MB
Using bcm_xgs_gmac-0 device
TFTP from server 192.168.1.251; our IP address is 192.168.1.1
Filename 'SPS09010jufi.bin'.
Load address: 0x70000000
Loading: #################################################################
############
1.1 MiB/s
done
Bytes transferred = 91769159 (5784947 hex)

Copying U-Boot image (mnz10126.bin) to Primary flash
sf erase 0x0 0xc0000
sf write 0x71cc0f34 0x0 0xc0000
device 0 offset 0x0, size 0xc0000
U-Boot update Done.

Copying FI image (SPS09010j.bin) to Primary flash
img_sz=0x1c00000 block_count = 0x1c

NAND erase.spread: device 0 offset 0x0, size 0x1c00000
Erasing at 0x1a00000 -- 100% complete.
OK

NAND write: device 0 offset 0x0, size 0x1c00000
29360128 bytes written: OK
FI-image update Done.
TFTP to Flash Done.
ICX7150-Boot>

[matt_radecic]

telnet@ICX7150-C12 Switch#show version
Copyright (c) Ruckus Networks, Inc. All rights reserved.
UNIT 1: compiled on Aug 8 2023 at 22:55:03 labeled as SPS08095m
(31457280 bytes) from Primary SPS08095m.bin (UFI)
SW: Version 08.0.95mT211
Compressed Primary Boot Code size = 786944, Version:10.1.26T225 (mnz10126)
Compiled on Tue Nov 29 12:43:26 2022

HW: Stackable ICX7150-C12-POE
==========================================================================
UNIT 1: SL 1: ICX7150-C12-2X1G POE 12-port Management Module
Serial #:FEK3820R2E2
Software Package: BASE_SOFT_PACKAGE
Current License: 2X1G
P-ASIC 0: type B160, rev 11 Chip BCM56160_B0
==========================================================================
UNIT 1: SL 2: ICX7150-2X1GC 2-port 2G Module
==========================================================================
UNIT 1: SL 3: ICX7150-2X10GF 2-port 20G Module
==========================================================================
1000 MHz ARM processor ARMv7 88 MHz bus
8 MB boot flash memory
2 GB code flash memory
1 GB DRAM
STACKID 1 system uptime is 2 day(s) 3 hour(s) 7 minute(s) 27 second(s)
The system started at 00:00:38 GMT+00 Wed Aug 09 2023
The system : started=factory reset reloaded=by "factory reset"

telnet@ICX7150-C12 Switch#show flash
Stack unit 1:
NAND Type: Micron NAND 2GiB (x 1)
Compressed Pri Code size = 31457280, Version:08.0.95mT211 (SPS08095m.bin)
Compressed Sec Code size = 31457280, Version:08.0.95mT211 (SPS08095m.bin)
Compressed Pri Boot Code size = 786944, Version:10.1.26T225 (mnz10126)
Compressed Sec Boot Code size = 786944, Version:10.1.26T225 (mnz10126)
Code Flash Free Space = 1207894016
telnet@ICX7150-C12 Switch#

[Imran_ruckus]

Hi @matt_radecic 

Thank you for the response and the information.

Please execute the below commands and check if you get the same error.

write memory

reload

Please check and let me know.

Thank you again for your patience and understanding.

Best regards,

Imran Sanadi

RUCKUS Lennar Home Community.

[matt_radecic]

the switch is in service and operational. not sure how a write memory will update any package installations as it is looking for specifical files (even after a factory reset)

i can try a reboot after hours to see if anything recovers, but what other options are there before that.  I will not have a direct terminal connection to capture logs, so from a telnet connection, i would not see any boot up messages. 

[Imran_ruckus]

Hi @matt_radecic 

Greetings!!!

Thank you for the response.

Please note, that we have some major changes in the DHCP for the latest versions. When the Switch acts as a DHCP server, is the reason you may see some directory files missing. For Lennar users, we do not use Switch DHCP as a server since the Switch gets an IP address assigned to it by the ISP Router.

You can ignore those errors. Since it doesn't impact the normal operation.

Thank you again for your patience and understanding.

Best regards,

Imran Sanadi

RUCKUS Lennar Home Community.