topic Re: POE Fatal message is flooding console on ICS7150-c12 switch in RUCKUS Support for Lennar Homes

POE Fatal message is flooding console on ICS7150-c12 switch

larry_speth — Wed, 04 Jan 2023 16:26:20 GMT

I have a Lennar smart house with a Ruckus 7150-c12 switch and a Ruckus R510 access point. After connecting to the console port I found the 7150 was continually send the message "PoE Fatal: Vmain (48V) fault has forced all ports on slot 0 to lose power". I was not experiencing any problems with any of the POE devices I have. I am researching a periodic problem of losing the internet connection for a few minutes and then it comes back. I tried a reset but after rebooting the 7150 resumes the continual error messages. I then tried a reset with a power on which my manual says should do a factory reset but again after reboot the error messages continue. Note that I have removed all cables other than the console cable from the 7150. I would like to update the firmware but can't get any commands into the console because of the stream of error msgs.

Re: POE Fatal message is flooding console on ICS7150-c12 switch

Yarenis — Wed, 04 Jan 2023 17:04:45 GMT

Hi @larry_speth

Thank you for the details provided, please gain access to the Switch via console, SSH or Telnet and share the next commands show version (Copy and paste in to the CLI is the best option instead to type due to the error), please provide the screenshot of putty session with the error and the results of the commands.

Best regards,

Yarenis Hernández.

Technical Support Engineer | L2 TAC Wired | Lennar Home Community

Re: POE Fatal message is flooding console on ICS7150-c12 switch

larry_speth — Wed, 04 Jan 2023 18:06:09 GMT

I am using Putty on a Windows machine. I have attached (I don't see any options to attach) a screen grab to show the error msg. It would not grab the screen without having to close the session to freeze it. I can't get any commands to take unless the error messages stop coming. It may be I just don't know enough about putty to get the messages to stop coming.
Larry

Re: POE Fatal message is flooding console on ICS7150-c12 switch

Yarenis — Wed, 04 Jan 2023 18:50:41 GMT

Hi @larry_speth

Sure, if you cannot run any commands please share the snipping, will be matter to copy and paste, or attached using the next option.

Best regards,

Yarenis Hernández.

Technical Support Engineer | L2 TAC Wired | Lennar Home Community

Re: POE Fatal message is flooding console on ICS7150-c12 switch

larry_speth — Wed, 04 Jan 2023 19:18:29 GMT

I used the Putty log to record what happens when I go through a reset:

=~=~=~=~=~=~=~=~=~=~=~= PuTTY log 2023.01.04 11:03:53 =~=~=~=~=~=~=~=~=~=~=~=
PoE Fatal: Vmain (48V) fault has forced all ports on slot 0 to lose power.
PoE Fatal: Vmain (48V) fault has forced all ports on slot 0 to lose power.
PoE Fatal: Vmain (48V) fault has forced all ports on slot 0 to lose power.
PoE Fatal: Vmain (48V) fault has forced all ports on slot 0 to lose power.
PoE Fatal: Vmain (48V) fault has forced all ports on slot 0 to lose power.
PoE Fatal: Vmain (48V) fault has forced all ports on slot 0 to lose power.
PoE Fatal: Vmain (48V) fault has forced all ports on slot 0 to lose power.
PoE Fatal: Vmain (48V) fault has forced all ports on slot 0 to lose power.
PoE Fatal: Vmain (48V) fault has forced all ports on slot 0 to lose power.
PoE Fatal: Vmain (48V) fault has forced all ports on slot 0 to lose power.
PoE Fatal: Vmain (48V) fault has forced all ports on slot 0 to lose power.
PoE Fatal: Vmain (48V) fault has forced all ports on slot 0 to lose power.
PoE Fatal: Vmain (48V) fault has forced all ports on slot 0 to lose power.
PoE Fatal: Vmain (48V) fault has forced all ports on slot 0 to lose power.
PoE Fatal: Vmain (48V) fault has forced all ports on slot 0 to lose power.
PoE Fatal: Vmain (48V) fault has forced all ports on slot 0 to lose power.
PoE Fatal: Vmain (48V) fault has forced all ports on slot 0 to lose power.
PoE Fatal: Vmain (48V) fault has forced all ports on slot 0 to lose power.
PoE Fatal: Vmain (48V) fault has forced all ports on slot 0 to lose power.
PoE Fatal: Vmain (48V) fault has forced all ports on slot 0 to lose power.
PoE Faô

Brocade Bootloader: 10.1.09T225 (Feb 17 2017 - 16:15:27 -0800)

Booted from partition 1
DRAM: Validate Shmoo parameters stored in flash ..... OK

ICX7150-12 (POE), PVT1
SYS CPLD VER: 0x4 Released Ver: 0x3
Writing Shmoo values into flash .....

device 0 offset 0x0, size 0xc0000
Enter 'b' to stop at boot monitor: 3 2 1 0
device 0 offset 0x0, size 0xc0000
bootdelay: ===
Booting image from Primary

NAND read: device 0 offset 0x0, size 0x2000000
33554432 bytes read: OK
## Loading kernel from FIT Image at 70000200 ...
Using 'conf@1' configuration
Trying 'kernel@2' kernel subimage
Description: Brocade Linux MN
Type: Kernel Image
Compression: lzma compressed
Data Start: 0x70208e64
Data Size: 2137958 Bytes = 2 MiB
Architecture: ARM
OS: Linux
Load Address: 0x61008000
Entry Point: 0x61008000
Hash algo: crc32
Hash value: 2c7dfff2
## Loading kernel from FIT Image at 70000200 ...
Using 'conf@1' configuration
Trying 'kernel@2' kernel subimage
Description: Brocade Linux MN
Type: Kernel Image
Compression: lzma compressed
Data Start: 0x70208e64
Data Size: 2137958 Bytes = 2 MiB
Architecture: ARM
OS: Linux
Load Address: 0x61008000
Entry Point: 0x61008000
Hash algo: crc32
Hash value: 2c7dfff2
Verifying Hash Integrity ... crc32+ OK
## Loading ramdisk from FIT Image at 70000200 ...
Using 'conf@1' configuration
Trying 'ramdisk@1' ramdisk subimage
Description: Ramdisk -rootfs
Type: RAMDisk Image
Compression: lzma compressed
Data Start: 0x704197c8
Data Size: 19641528 Bytes = 18.7 MiB
Architecture: ARM
OS: Linux
Load Address: 0x00000000
Entry Point: 0x00000000
Hash algo: crc32
Hash value: 27dad9a7
Verifying Hash Integrity ... crc32+ OK
## Loading fdt from FIT Image at 70000200 ...
Using 'conf@1' configuration
Trying 'fdt@3' fdt subimage
Description: Flattened Device Tree blob - Minion.dtb
Type: Flat Device Tree
Compression: uncompressed
Data Start: 0x70417570
Data Size: 8605 Bytes = 8.4 KiB
Architecture: ARM
Hash algo: crc32
Hash value: ea68dce3
Verifying Hash Integrity ... crc32+ OK
Loading fdt from 0x70417570 to 0x65008000
Booting using the fdt blob at 0x65008000
Uncompressing Kernel Image ... OK
Using Device Tree in place at 65008000, end 6500d19c

Starting kernel ...

booting with 4.4 kernel
dd cmd: UBI
Config partition mounted
ubi1 error: ubi_io_read: error -74 (ECC error) while reading 1024000 bytes from PEB 232:24576, read 1024000 bytes
UBIFS error (ubi1:0 pid 554): ubifs_recover_leb: corrupt empty space LEB 11:16384, corruption starts at 5943
UBIFS error (ubi1:0 pid 554): ubifs_scanned_corruption: corruption at LEB 11:22327
UBIFS error (ubi1:0 pid 554): ubifs_scanned_corruption: first 8192 bytes from LEB 11:22327
UBIFS error (ubi1:0 pid 554): ubifs_recover_leb: LEB 11 scanning failed
mount: mounting ubi1:resources on /opt failed: Structure needs cleaning
Resource partition is mounted!!
dma_mem_base: 0x96000000, dma_mem_len: 0x8000000, warm_mem_base: 0x9f500000, warm_mem_len: 0xb00000
Creating TUN device
Starting the FastIron.
i2c-0 smbus iproc-smb-1 SMBus adapter 3
i2c-1 smbus iproc-smb-1 SMBus adapter 3
FIPS Disabled:PORT NOT DISABLED
platform type 90
OS>
Loaded Image SPS08060 from Pri Built on (UTC): Sat Feb 18 00:07:03 2017
Applying factory defaults..
INFO: startup config data is not available, try to read from backup
INFO: startup config data in the backup area is not available
set_board_level: gi_board_type = 116DMA pool size: 134217728
PHY Reset De-asserted
sw_pp_sdk_init:111 ToR init for Minion Board
SOC unit 0 attached to PCI device BCM56160_B0
Pre Parsing Config Data ...
INFO: empty config data in the primary area, try to read from backup
INFO: empty config data in the backup area also

Parsing Config Data ...
INFO: empty config data in the primary area, try to read from backup
INFO: empty config data in the backup area also

Hotplugger Daemon Initialized ..

System initialization completed...console going online.
Copyright (c) 1996-2017 Brocade Communications Systems, Inc. All rights reserved.
UNIT 1: compiled on Feb 17 2017 at 16:03:13 labeled as SPS08060
(23946048 bytes) from Primary
SW: Version 08.0.60T211
Compressed Boot-Monitor Image size = 786944, Version:10.1.09T225 (mnz10109)
Compiled on Sat Feb 18 00:15:43 2017

HW: Stackable ICX7150-C12-POE
==========================================================================
UNIT 1: SL 1: ICX7150-C12-2X10GR POE 12-port Management Module
Serial #:FEK3224N0DT
Current License:
P-ASIC 0: type B160, rev 11 Chip BCM56160_B0
==========================================================================
UNIT 1: SL 2: ICX7150-2X1GC 2-port 2G Module
==========================================================================
UNIT 1: SL 3: ICX7150-2X10GF 2-port 20G Module
==========================================================================
1000 MHz ARM processor ARMv7 88 MHz bus
8192 KB boot flash memory
2048 MB code flash memory
1024 MB DRAM
STACKID 1 system uptime is 5 second(s)
The system started at 02:33:56 GMT+00 Sat Jan 01 2000

The system : started=cold start

ICX7150-C12 Switch>
ICX7150-C12 Switch>
PS 1, Internal Power supply detected.
PS 1, Internal Power supply is up.
PoE Info: Adding new 54V capacity of 124000 mW, total capacity is 124000, total free capacity is 124000
PoE Info: PoE module 1 of Unit 1 on ports 1/1/1 to 1/1/12 detected. Initializing....
PoE Event Trace Log Buffer for 2000 log entries allocated
PoE Event Trace Logging enabled...
PoE Info: PoE module 1 of Unit 1 initialization is done.
PoE Fatal: Vmain (48V) fault has forced all ports on slot 0 to lose power.
PoE Fatal: Vmain (48V) fault has forced all ports on slot 0 to lose power.
PoE Fatal: Vmain (48V) fault has forced all ports on slot 0 to lose power.
PoE Fatal: Vmain (48V) fault has forced all ports on slot 0 to lose power.
PoE Fatal: Vmain (48V) fault has forced all ports on slot 0 to lose power.
PoE Fatal: Vmain (48V) fault has forced all ports on slot 0 to lose power.
PoE Fatal: Vmain (48V) fault has forced all ports on slot 0 to lose power.
PoE Fatal: Vmain (48V) fault has forced all ports on slot 0 to lose power.

I could only discern that the level is SPS08060 and there appears to be some problems with UBIFS errors. Hope its more familiar to you.

Larry

Re: POE Fatal message is flooding console on ICS7150-c12 switch

Yarenis — Wed, 04 Jan 2023 19:43:20 GMT

Hi @ larry_speth

Correct there is a defect in that old code, however can be fixed by upgrading the Switch to the code 08095g UFI, so in order to perform that task lets disable PoE and the log messages, then upgrade, so you will be able to perform the upgrade with not issues.

1: Disabling PoE

enable

configure terminal

interface ethernet 1/1/1 to 1/1/13

no inline power

exit

2:Disabling log messages

configure terminal

no logging console

write memory

https://docs.ruckuswireless.com/fastiron/08.0.60/fastiron-08060-commandref/GUID-D65874A7-EF61-4668-A8B6-07829BD1AE45.html

3:Please perform an upgrade to the code 08095g UFI and upgrade both partitions primary and secondary with the switching code file named SPS08095gufi.bin, take into consideration that is the Switch is running a code lower than 08080, so You need to upgrade twice, from the code you are to 08080f and then the recommended code 08095g UFI.

08060>08080f>8095g UFI

How to Upgrade a Switch ICX-7150-C12P using a USB flash drive

https://community.ruckuswireless.com/t5/RUCKUS-Support-for-Lennar-Homes/How-to-Upgrade-a-Switch-ICX-7150-C12P-using-a-USB-flash-drive/td-p/46781

How to upgrade a Switch ICX7150-C12P via TFTP

https://community.ruckuswireless.com/t5/RUCKUS-Support-for-Lennar-Homes/How-to-Upgrade-a-Switch-ICX-7150-C12P-via-TFTP/td-p/47187

4:Enabling back PoE:

enable

configure terminal

interface ethernet 1/1/1 to 1/1/13

inline power

exit

https://docs.commscope.com/en-US/bundle/fastiron-08090-managementguide/page/GUID-B427DDA2-C8F4-40F7-A100-9A8E7DC55156.html

Best regards,

Yarenis Hernández.

Technical Support Engineer | L2 TAC Wired | Lennar Home Community

Re: POE Fatal message is flooding console on ICS7150-c12 switch

larry_speth — Wed, 04 Jan 2023 21:14:30 GMT

When I try to enter "enable" I get "no password has been assigned yet...." and all commands after that are invalid or broken into little pieces due to the stream of error messages. Why does the factory reset not put the unit back to its original state?

Re: POE Fatal message is flooding console on ICS7150-c12 switch

Yarenis — Wed, 04 Jan 2023 21:24:05 GMT

Hi @larry_speth

Please share the putty session error, after you run the enable commands, will be expected you get the no password message, because this device does not have set a password.

Best regards,

Yarenis Hernández.

Technical Support Engineer | L2 TAC Wired | Lennar Home Community

Re: POE Fatal message is flooding console on ICS7150-c12 switch

larry_speth — Thu, 05 Jan 2023 23:58:35 GMT

Thank you for your help but this has taken way too much of my time to try to fix the 7150. I have replaced the 7150 with another vendor's switch and have turned off the r510 and converted to a mesh system. Your company really needs to move away from the CLI control and switch to web control like most of the other network vendors.

Re: POE Fatal message is flooding console on ICS7150-c12 switch

Yarenis — Fri, 06 Jan 2023 03:34:19 GMT

Hi @ larry_speth

The CLI Option is one of the many ways to gain access to the Switch, however, due to the error the only option to recover the Switch will be via console access.

Best regards,

Yarenis Hernández.

Technical Support Engineer | L2 TAC Wired | Lennar Home Community