I broke a 7450 stack

[clayton_taverni]

We had a stack set up with a 7650 and 5 7450s.  There's an issue with the primary and secondary flash on one of the 7450s and, in trying to fix it, I rebooted the stack.  Now, the last two switches have been dropped from the stack and I would like some help getting them back.

SSH@STACK#show spx
T=4h39m10.5: alone: standalone, 😧 dynamic cfg, S: static
ID   Type          Role    Mac Address    Pri State   Comment                   
1  S ICX7650-48ZP  alone   d4c1.9e14.78c3   0 local   Ready
21 S ICX7450-48P   spx-pe  d4c1.9e08.2440 N/A remote  Ready
22 S ICX7450-48P   spx-pe  d4c1.9e07.f680 N/A remote  Ready
23 S ICX7450-48P   spx-pe  d4c1.9e08.1f00 N/A remote  Ready
24 S ICX7450-48P   spx-pe  0000.0000.0000 N/A reserve 
25 S ICX7450-48P   spx-pe  0000.0000.0000 N/A reserve 

                                                                               
     +---+                                                                     
  3/1| 1 |3/3                                                                  
     +---+                                                                     
            +----+        +----+        +----+                                 
  1/3/2--3/1| 21 |4/1--3/1| 22 |4/1--3/1| 23 |4/1-                             
            +----+        +----+        +----+                                 
                                                                               
This is from before the reboot:

SSH@STACK#show flash
Stack unit 1:
  Compressed Pri Code size = 56904124, Version:08.0.80bT233 (TNR08080b.bin)
  Compressed Sec Code size = 56904124, Version:08.0.80bT233 (TNR08080b.bin)
  Compressed Boot-Monitor Image size = 1573376, Version:10.1.13T235
  Code Flash Free Space = 2802184192
SPX unit 21: 
  Compressed Pri Code size = 29815996, Version:08.0.80bT213 (SPR08080b.bin)
  Compressed Sec Code size = 29815996, Version:08.0.80bT213 (SPR08080b.bin)
  Compressed Boot-Monitor Image size = 786432, Version:10.1.13T215
  Code Flash Free Space = 1697820672
SPX unit 22: 
  Compressed Pri Code size = 29815996, Version:08.0.80bT213 (SPR08080b.bin)
  Compressed Sec Code size = 29815996, Version:08.0.80bT213 (SPR08080b.bin)
  Compressed Boot-Monitor Image size = 786432, Version:10.1.13T215
  Code Flash Free Space = 1697820672
SPX unit 23: 
  Compressed Pri Code size = 29815996, Version:08.0.80bT213 (SPR08080b.bin)
  Compressed Sec Code size = 29815996, Version:08.0.80bT213 (SPR08080b.bin)
  Compressed Boot-Monitor Image size = 786432, Version:10.1.13T215
  Code Flash Free Space = 1697816576
SPX unit 24: 
  Compressed Pri Code size = -1523736068, Version:08.0.80T213 (SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0�-��08.0.80T213)
  Compressed Sec Code size = -1523736068, Version:08.0.80T213 (SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0�-��10.1.12T215)
  Compressed Boot-Monitor Image size = 786944, Version:10.1.12T215
  Code Flash Free Space = 1760612352
SPX unit 25: 
  Compressed Pri Code size = 29815996, Version:08.0.80bT213 (SPR08080b.bin)
  Compressed Sec Code size = 29815996, Version:08.0.80bT213 (SPR08080b.bin)
  Compressed Boot-Monitor Image size = 786432, Version:10.1.13T215
  Code Flash Free Space = 1697816576



Suggestions?

[netwizz]

Were you able to get it resolved?  If not the next step is to fix each of the switches stuck in boot monitor.  I would probably remove the stacking cables from them provided you don't have the stack ring broken in two places.

Then I would recover the fimware until the unit is bootable.  At that point I would do a stack unconfigure me.

Finally, I would place back into the stack, ensure proper speed settings for the Stacking Cables... then enable stacking and hitless failover.  Most likely the unit would reboot and re-join.

At any rate have the order and all the mac addresses.

You can re-run the stack secure-setup if necessary and re-number as well.

Ensure you have a stack backup of the configuration.

Regardless, if worse comes to worst, you should be able to rebuild it within an hour or two from scratch.

[clayton_taverni]

Thanks.  The problem switch is stuck in boot mode.  We directly connected a TFTP server and were able to upload the proper firmware but when we rebooted, we got this error. 

"Wrong Image Format for bootm command"
"ERROR: can't get kernel image!"

I'm still waiting for Ruckus Support to get back to me about this one.

[hashim_bharooc1]

Hi Clayton,
Happy New Year.

This will require an RMA.

I  Pinged the case owner for 00881158.

They will RMA it with Failure Analysis, so we can figure out what went wrong.

Thanks
Hashim
 

[netwizz]

Looking through my console logs, I did a recovery on or about August 2nd, 2018 on a 7150.  It should be nearly identical... only you need to send the appropriate firmware for a 7450 and not the firmware I sent for a 64XX...



NOTE:  I used Tftpd64 by Ph. Jounin.  This doesn't mean you need to use the same TFTP server, but I can verify this one works flawlessly.  Make sure it lists the Server Interface for the Network adapter you test ping later in this exercise, so we know it is reachable.  Also make certain the "Current Directory" is appropriate for your TFTP-Root with your image in the root directory.


Remove the switch to your desk and try the following (but with a compatible image for your model) AND the current boot monitor:


ICX64XX-boot>> printenv

You will get something like this only different:

baudrate=9600
ipaddr=10.115.142.82
serverip=10.115.1.80
netmask=255.255.255.0
gatewayip=10.115.128.1
uboot=kxz10105
image_name=/foundry/FGS/os/ICX64S08010h.bin
ver=10.1.05T310 (Mar 19 2015 - 16:39:59)

Now try this.  In my example, uboot is my TFTP server IP.  I do not think the serverip is relevant; I don't know what it does.  It seems the switch uses ipaddr!  Set these to work with your environment....


ICX64XX-boot>> setenv ipaddr 10.1.2.24
ICX64XX-boot>> setenv netmask 255.255.248.0
ICX64XX-boot>> setenv gatewayip 10.1.2.1
ICX64XX-boot>> set uboot 10.1.2.4
ICX64XX-boot>> set image_name ICX64S08030s.bin

ICX64XX-boot>> printenv
baudrate=9600
ipaddr=10.1.2.24
serverip=10.115.1.80
netmask=255.255.248.0
gatewayip=10.1.2.1
uboot=10.1.2.4
image_name=ICX64S08030s.bin





Save your environment

ICX64XX-boot>> saveenv


Connect the management interface to somewhere it can reach your TFTP server... and verify it can reach by pinging the server.  NOTE:  This is the ONLY interface on the switch that will work for a recovery!  On many switches this is near the console port!

ICX64XX-boot>> ping 10.1.2.4
Using egiga0 device
host 10.1.2.4 is alive


Get It! Pull the Image from the TFTP server:


ICX64XX-boot>> update_secondary
Using egiga0 device
TFTP from server 10.1.2.4; our IP address is 10.1.2.24
Download Filename 'ICX64S08030s.bin'.
Load address: 0x3000000
Download to address: 0x3000000
Loading: *%#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
################################################################
done
Bytes transferred = 8558924 (82994c hex)
prot off f9080000 f9ffffff
........................................................................................................................................................................................................................................................
Un-Protected 248 sectors
erase f9080000 f9ffffff

.............................................................
.................................................................
.................................................................
.........................................................
Erased 248 sectors
copying image to flash, it will take sometime...
sflash write 3000000 1080000 f80000
TFTP to Flash Done.
ICX64XX-boot>> <
ICX64XX-boot>>
ICX64XX-boot>> 
ICX64XX-boot>> 


BOOT the FastIron Image:


ICX64XX-boot>> boot_secondary
Booting image from Secondary
## Booting image at 00007fc0 ...
   Created:      2018-05-30  14:30:33 UTC
   Data Size:    8558348 Bytes =  8.2 MB
   Load Address: 00008000
   Entry Point:  00008000
   Verifying Checksum ... OK
OK

Starting kernel in BE mode ...
Uncompressing Image...................................................................................................................................................................................................................................................................................................................................................................... done, booting the kernel.

Config partition mounted.
Creating TUN device
Starting the FastIron.
FIPS Disabled:PORT NOT DISABLED
platform type 49
OS>Unable to set the kernel wall time 
Starting Main Task .INFO: startup config data is not available, try to read from backup
INFO: startup config data in the backup area is not available
CPSS DxCh Version: cpss3.4p1 release
Pre Parsing Config Data ...
INFO: empty config data in the primary area, try to read from backup
INFO: empty config data in the backup area also

Parsing Config Data ...
INFO: empty config data in the primary area, try to read from backup
INFO: empty config data in the backup area also

System initialization completed...console going online.

  Copyright (c) 1996-2016 Brocade Communications Systems, Inc. All rights reserved.
    UNIT 1: compiled on May 30 2018 at 07:29:29 labeled as ICX64S08030s
(8558924 bytes) from Secondary ICX64S08030s.bin
        SW: Version 08.0.30sT311 
  Boot-Monitor Image size = 786944, Version:10.1.05T310 (10)
  HW: Stackable ICX6450-C12-PD
==========================================================================
UNIT 1: SL 1: ICX6450C 12-port-PD Management Module
  Serial  #: REDACTED
  License: BASE_SOFT_PACKAGE   (LID: eviHKJMnFNi)
  P-ENGINE  0: type DEF0, rev 01
==========================================================================
UNIT 1: SL 2: ICX6450C-Copper 2port 2G Module
==========================================================================
UNIT 1: SL 3: ICX6450C-Fiber 2port 2G Module
==========================================================================
  800 MHz ARM processor ARMv5TE, 400 MHz bus
65536 KB flash memory
  512 MB DRAM
STACKID 1  system uptime is 6 second(s) 
The system : started=warm start reloaded=by "reload"

ICX6450-C12PD Switch>
Stack unit 1 PS 1, Internal Power supply detected and up.

Stack unit 1 PS 1, Internal Power supply detected and up.
PoE: Stack unit 1 PS 1, Internal Power supply  with 68000 mwatts capacity is up
PoE Info: Adding new 54V capacity of 68000 mW, total capacity is 68000, total free capacity is 68000
PoE Info: PoE module 1 of Unit 1 on ports 1/1/1 to 1/1/4 detected. Initializing....
PoE Event Trace Log Buffer for 2000 log entries allocated
PoE Event Trace Logging enabled...
PoE Info: PoE module 1 of Unit 1 initialization is done. 
^C

ICX6450-C12PD Switch>

[netwizz]

Or you can RMA it, lol.  They posted that when I was writing the above.   Honestly, I don't know what happened.  I never had a situation where both the primary and secondary were corrupted at the same time, but I did a test recovery on an old device just as an exercise once.

If one or the other firmware is good, you might merely try...  It just seems unlikely both flash slots would be corrupt at the same time.  Regardless, it is my opinion if the boot-monitor is undamaged, I would think the switch is recoverable.

ICX64XX-boot>> boot_primary

or

ICX64XX-boot>> boot_secondary


That said, if they offer an RMA, that may be the easier path.