I broke a 7450 stack

[clayton_taverni]

We had a stack set up with a 7650 and 5 7450s.  There's an issue with the primary and secondary flash on one of the 7450s and, in trying to fix it, I rebooted the stack.  Now, the last two switches have been dropped from the stack and I would like some help getting them back.

SSH@STACK#show spx
T=4h39m10.5: alone: standalone, 😧 dynamic cfg, S: static
ID   Type          Role    Mac Address    Pri State   Comment                   
1  S ICX7650-48ZP  alone   d4c1.9e14.78c3   0 local   Ready
21 S ICX7450-48P   spx-pe  d4c1.9e08.2440 N/A remote  Ready
22 S ICX7450-48P   spx-pe  d4c1.9e07.f680 N/A remote  Ready
23 S ICX7450-48P   spx-pe  d4c1.9e08.1f00 N/A remote  Ready
24 S ICX7450-48P   spx-pe  0000.0000.0000 N/A reserve 
25 S ICX7450-48P   spx-pe  0000.0000.0000 N/A reserve 

                                                                               
     +---+                                                                     
  3/1| 1 |3/3                                                                  
     +---+                                                                     
            +----+        +----+        +----+                                 
  1/3/2--3/1| 21 |4/1--3/1| 22 |4/1--3/1| 23 |4/1-                             
            +----+        +----+        +----+                                 
                                                                               
This is from before the reboot:

SSH@STACK#show flash
Stack unit 1:
  Compressed Pri Code size = 56904124, Version:08.0.80bT233 (TNR08080b.bin)
  Compressed Sec Code size = 56904124, Version:08.0.80bT233 (TNR08080b.bin)
  Compressed Boot-Monitor Image size = 1573376, Version:10.1.13T235
  Code Flash Free Space = 2802184192
SPX unit 21: 
  Compressed Pri Code size = 29815996, Version:08.0.80bT213 (SPR08080b.bin)
  Compressed Sec Code size = 29815996, Version:08.0.80bT213 (SPR08080b.bin)
  Compressed Boot-Monitor Image size = 786432, Version:10.1.13T215
  Code Flash Free Space = 1697820672
SPX unit 22: 
  Compressed Pri Code size = 29815996, Version:08.0.80bT213 (SPR08080b.bin)
  Compressed Sec Code size = 29815996, Version:08.0.80bT213 (SPR08080b.bin)
  Compressed Boot-Monitor Image size = 786432, Version:10.1.13T215
  Code Flash Free Space = 1697820672
SPX unit 23: 
  Compressed Pri Code size = 29815996, Version:08.0.80bT213 (SPR08080b.bin)
  Compressed Sec Code size = 29815996, Version:08.0.80bT213 (SPR08080b.bin)
  Compressed Boot-Monitor Image size = 786432, Version:10.1.13T215
  Code Flash Free Space = 1697816576
SPX unit 24: 
  Compressed Pri Code size = -1523736068, Version:08.0.80T213 (SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0�-��08.0.80T213)
  Compressed Sec Code size = -1523736068, Version:08.0.80T213 (SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0�-��10.1.12T215)
  Compressed Boot-Monitor Image size = 786944, Version:10.1.12T215
  Code Flash Free Space = 1760612352
SPX unit 25: 
  Compressed Pri Code size = 29815996, Version:08.0.80bT213 (SPR08080b.bin)
  Compressed Sec Code size = 29815996, Version:08.0.80bT213 (SPR08080b.bin)
  Compressed Boot-Monitor Image size = 786432, Version:10.1.13T215
  Code Flash Free Space = 1697816576



Suggestions?

[clayton_taverni]

NETWizz, when we reboot the switch, it shows a message that the 10.1.13 (spz10113) is installed but it's expecting spz10112.  This confuses me.

All the other switches in the stack have spz10113 installed.

[clayton_taverni]

Thanks Michael but the switch is booting okay now with the proper firmware installed and no configuration.  Now I just need to get it back into the stack.

[netwizz]

Awesome; I am glad it helped.

TAKE A BACKUP OF YOUR STACK CONFIG FIRST!!!



First I would do a "stack unconfigure me" or perhaps an "stack unconfigure all"... and a reload.  Then I would do an "erase startup-config" followed by a reload.


If it says it is expecting spz10112, I would probably send that over and reload it to see if that message goes away. Then you can put back the bootrom of your choice, and it should go back through the upgrade procedure to upgrade the bootmonitor.

At any rate, it is a bit different being you are running the router firmware SPR08080B I think you said.

On a blank switch, we need to put an IP on an interface that is within the same subnet as your TFTP server or if that TFTP server is on a different subnet, you need to also put a next hop to point to your LAN's default-gateway.

Basically choose an interface like Ethernet 1/1/48 or whatever and put an IP on it within your network on a Layer-3 Interface.

switch>
switch# conf t
switch(config)# int e 1/1/48
switch(config-if-e1000-1/1/48)# ip add 10.1.2.3/24
switch(config-if-e1000-1/1/48)# enable

Not needed it the TFTP server is on the SAME LAN... /24 above is for the subnet mask 255.255.255.0 (adjust accordingly to your LAN subnet)

switch(config-if-e1000-1/1/48)# exit
switch(config)# ip route 0.0.0.0/0 10.1.2.1


To Downgrade to 10.1.12 boot monitor that it is expecting:

switch# copy tftp flash 10.1.2.4 spz10112.bin bootrom



Please do a "show flash" to verify it is present before reloading.  Sometimes it takes a minute after the copy completes before it is really ready!



****************

Once everything is looking great, you can re-add the device to your stack.

Basically, you would put it back in and reconnect the cables.  It is best if it has a matching bootrom and firmware!

Once you connect the stacking cables, if you do not get messages on the terminal that they changed state to UP, you would need to adjust the speed.  I do not think this was an issue on the 7450's I dealt with though.


At this point

The entire stack should probably run


stack (config)# hitless-failover enable

Switch you add:
switch(config)# stack enable


Please do this in a maintenance window.  If you have hitless-failover, only the new device should reboot.  Otherwise the entire stack will likely reboot.


Next, you probably want the MAC addresses of each switch.... top to bottom; since, you want them numbered in order...

Do a "sh stack"

If all is well you will see a ring topology shown.
You are also looking for:  "Standby u2 - protocols ready, can failover or manually switch over"

You really do not want to make any changes until everything comes into convergence within the stack.  At that point, I would do a "wr mem"


Next to renumber, rerun your "stack secure-setup"

If you have your list of MAC addresses, it can walk you through renumbering all the switches.


Once complete, you want to validate communications


SSH@SWITCH#sh stack connection
Probing the topology. Please wait ...
SSH@SWITCH#
    active       standby
     +---+        +---+
 -4/1| 1 |3/1--4/1| 2 |3/1-
 |   +---+        +---+   |
 |                        |
 |------------------------|

trunk probe results: 2 links
Link 1: u1 -- u2, num=1
  1: 1/3/1 (P0) <---> 2/4/1 (P1)
Link 2: u1 -- u2, num=1
  1: 1/4/1 (P1) <---> 2/3/1 (P0)
CPU to CPU packets are fine between 2 units.



Basically, you are looking that the CPU to CPU packets are okay all around the stack.


If all is great, "wr mem"


At any rate, after a renumber, you may have to change some VLAN memberships etc.  Ultimately, though this should be readily repairable.

[clayton_taverni]

Sorry, I forgot to post this update yesterday:

I have no idea how it happened but when I went to console into the problem switch earlier today, I didn't get the boot-mode prompt, I got the new switch prompt (empty running config).  I ran the "show version" and both the primary and secondary flashes had the correct firmware.  I guess whatever I did on Friday with the firmware and the bootrom took a while to process.

I had backed up the running-config last week and I managed to download it into the starting-config (initially there was no starting-config.txt file to download into until I made some small change to the empty running-config and wrote it to memory) but after a reload, I still had an empty running-config.  Maybe the starting-config for the stack wasn't going to work as the starting-config for the switch.

[clayton_taverni]

I think we're going to RMA the problem switch.  We went through the procedures for having the switch "discovered" by the stack and it never worked.  It seems to be that whatever the zero-input process or the spx configuration save in the switch's flash memory that configures it as a Port Extension is lost every time it reboots.

Thanks everyone for all your input.