I broke a 7450 stack
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-26-2018 03:27 PM
We had a stack set up with a 7650 and 5 7450s. There's an issue with the primary and secondary flash on one of the 7450s and, in trying to fix it, I rebooted the stack. Now, the last two switches have been dropped from the stack and I would like some help getting them back.
SSH@STACK#show spx
T=4h39m10.5: alone: standalone, 😧 dynamic cfg, S: static
ID Type Role Mac Address Pri State Comment
1 S ICX7650-48ZP alone d4c1.9e14.78c3 0 local Ready
21 S ICX7450-48P spx-pe d4c1.9e08.2440 N/A remote Ready
22 S ICX7450-48P spx-pe d4c1.9e07.f680 N/A remote Ready
23 S ICX7450-48P spx-pe d4c1.9e08.1f00 N/A remote Ready
24 S ICX7450-48P spx-pe 0000.0000.0000 N/A reserve
25 S ICX7450-48P spx-pe 0000.0000.0000 N/A reserve
+---+
3/1| 1 |3/3
+---+
+----+ +----+ +----+
1/3/2--3/1| 21 |4/1--3/1| 22 |4/1--3/1| 23 |4/1-
+----+ +----+ +----+
This is from before the reboot:
SSH@STACK#show flash
Stack unit 1:
Compressed Pri Code size = 56904124, Version:08.0.80bT233 (TNR08080b.bin)
Compressed Sec Code size = 56904124, Version:08.0.80bT233 (TNR08080b.bin)
Compressed Boot-Monitor Image size = 1573376, Version:10.1.13T235
Code Flash Free Space = 2802184192
SPX unit 21:
Compressed Pri Code size = 29815996, Version:08.0.80bT213 (SPR08080b.bin)
Compressed Sec Code size = 29815996, Version:08.0.80bT213 (SPR08080b.bin)
Compressed Boot-Monitor Image size = 786432, Version:10.1.13T215
Code Flash Free Space = 1697820672
SPX unit 22:
Compressed Pri Code size = 29815996, Version:08.0.80bT213 (SPR08080b.bin)
Compressed Sec Code size = 29815996, Version:08.0.80bT213 (SPR08080b.bin)
Compressed Boot-Monitor Image size = 786432, Version:10.1.13T215
Code Flash Free Space = 1697820672
SPX unit 23:
Compressed Pri Code size = 29815996, Version:08.0.80bT213 (SPR08080b.bin)
Compressed Sec Code size = 29815996, Version:08.0.80bT213 (SPR08080b.bin)
Compressed Boot-Monitor Image size = 786432, Version:10.1.13T215
Code Flash Free Space = 1697816576
SPX unit 24:
Compressed Pri Code size = -1523736068, Version:08.0.80T213 (SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0�-��08.0.80T213)
Compressed Sec Code size = -1523736068, Version:08.0.80T213 (SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0�-��10.1.12T215)
Compressed Boot-Monitor Image size = 786944, Version:10.1.12T215
Code Flash Free Space = 1760612352
SPX unit 25:
Compressed Pri Code size = 29815996, Version:08.0.80bT213 (SPR08080b.bin)
Compressed Sec Code size = 29815996, Version:08.0.80bT213 (SPR08080b.bin)
Compressed Boot-Monitor Image size = 786432, Version:10.1.13T215
Code Flash Free Space = 1697816576
Suggestions?
SSH@STACK#show spx
T=4h39m10.5: alone: standalone, 😧 dynamic cfg, S: static
ID Type Role Mac Address Pri State Comment
1 S ICX7650-48ZP alone d4c1.9e14.78c3 0 local Ready
21 S ICX7450-48P spx-pe d4c1.9e08.2440 N/A remote Ready
22 S ICX7450-48P spx-pe d4c1.9e07.f680 N/A remote Ready
23 S ICX7450-48P spx-pe d4c1.9e08.1f00 N/A remote Ready
24 S ICX7450-48P spx-pe 0000.0000.0000 N/A reserve
25 S ICX7450-48P spx-pe 0000.0000.0000 N/A reserve
+---+
3/1| 1 |3/3
+---+
+----+ +----+ +----+
1/3/2--3/1| 21 |4/1--3/1| 22 |4/1--3/1| 23 |4/1-
+----+ +----+ +----+
This is from before the reboot:
SSH@STACK#show flash
Stack unit 1:
Compressed Pri Code size = 56904124, Version:08.0.80bT233 (TNR08080b.bin)
Compressed Sec Code size = 56904124, Version:08.0.80bT233 (TNR08080b.bin)
Compressed Boot-Monitor Image size = 1573376, Version:10.1.13T235
Code Flash Free Space = 2802184192
SPX unit 21:
Compressed Pri Code size = 29815996, Version:08.0.80bT213 (SPR08080b.bin)
Compressed Sec Code size = 29815996, Version:08.0.80bT213 (SPR08080b.bin)
Compressed Boot-Monitor Image size = 786432, Version:10.1.13T215
Code Flash Free Space = 1697820672
SPX unit 22:
Compressed Pri Code size = 29815996, Version:08.0.80bT213 (SPR08080b.bin)
Compressed Sec Code size = 29815996, Version:08.0.80bT213 (SPR08080b.bin)
Compressed Boot-Monitor Image size = 786432, Version:10.1.13T215
Code Flash Free Space = 1697820672
SPX unit 23:
Compressed Pri Code size = 29815996, Version:08.0.80bT213 (SPR08080b.bin)
Compressed Sec Code size = 29815996, Version:08.0.80bT213 (SPR08080b.bin)
Compressed Boot-Monitor Image size = 786432, Version:10.1.13T215
Code Flash Free Space = 1697816576
SPX unit 24:
Compressed Pri Code size = -1523736068, Version:08.0.80T213 (SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0�-��08.0.80T213)
Compressed Sec Code size = -1523736068, Version:08.0.80T213 (SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0SPR0�-��10.1.12T215)
Compressed Boot-Monitor Image size = 786944, Version:10.1.12T215
Code Flash Free Space = 1760612352
SPX unit 25:
Compressed Pri Code size = 29815996, Version:08.0.80bT213 (SPR08080b.bin)
Compressed Sec Code size = 29815996, Version:08.0.80bT213 (SPR08080b.bin)
Compressed Boot-Monitor Image size = 786432, Version:10.1.13T215
Code Flash Free Space = 1697816576
Suggestions?
19 REPLIES 19
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-03-2019 09:42 AM
NETWizz, when we reboot the switch, it shows a message that the 10.1.13 (spz10113) is installed but it's expecting spz10112. This confuses me.
All the other switches in the stack have spz10113 installed.
All the other switches in the stack have spz10113 installed.
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-03-2019 09:43 AM
Thanks Michael but the switch is booting okay now with the proper firmware installed and no configuration. Now I just need to get it back into the stack.
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-03-2019 11:07 AM
Awesome; I am glad it helped.
TAKE A BACKUP OF YOUR STACK CONFIG FIRST!!!
First I would do a "stack unconfigure me" or perhaps an "stack unconfigure all"... and a reload. Then I would do an "erase startup-config" followed by a reload.
If it says it is expecting spz10112, I would probably send that over and reload it to see if that message goes away. Then you can put back the bootrom of your choice, and it should go back through the upgrade procedure to upgrade the bootmonitor.
At any rate, it is a bit different being you are running the router firmware SPR08080B I think you said.
On a blank switch, we need to put an IP on an interface that is within the same subnet as your TFTP server or if that TFTP server is on a different subnet, you need to also put a next hop to point to your LAN's default-gateway.
Basically choose an interface like Ethernet 1/1/48 or whatever and put an IP on it within your network on a Layer-3 Interface.
switch>
switch# conf t
switch(config)# int e 1/1/48
switch(config-if-e1000-1/1/48)# ip add 10.1.2.3/24
switch(config-if-e1000-1/1/48)# enable
Not needed it the TFTP server is on the SAME LAN... /24 above is for the subnet mask 255.255.255.0 (adjust accordingly to your LAN subnet)
switch(config-if-e1000-1/1/48)# exit
switch(config)# ip route 0.0.0.0/0 10.1.2.1
To Downgrade to 10.1.12 boot monitor that it is expecting:
switch# copy tftp flash 10.1.2.4 spz10112.bin bootrom
Please do a "show flash" to verify it is present before reloading. Sometimes it takes a minute after the copy completes before it is really ready!
****************
Once everything is looking great, you can re-add the device to your stack.
Basically, you would put it back in and reconnect the cables. It is best if it has a matching bootrom and firmware!
Once you connect the stacking cables, if you do not get messages on the terminal that they changed state to UP, you would need to adjust the speed. I do not think this was an issue on the 7450's I dealt with though.
At this point
The entire stack should probably run
stack (config)# hitless-failover enable
Switch you add:
switch(config)# stack enable
Please do this in a maintenance window. If you have hitless-failover, only the new device should reboot. Otherwise the entire stack will likely reboot.
Next, you probably want the MAC addresses of each switch.... top to bottom; since, you want them numbered in order...
Do a "sh stack"
If all is well you will see a ring topology shown.
You are also looking for: "Standby u2 - protocols ready, can failover or manually switch over"
You really do not want to make any changes until everything comes into convergence within the stack. At that point, I would do a "wr mem"
Next to renumber, rerun your "stack secure-setup"
If you have your list of MAC addresses, it can walk you through renumbering all the switches.
Once complete, you want to validate communications
SSH@SWITCH#sh stack connection
Probing the topology. Please wait ...
SSH@SWITCH#
active standby
+---+ +---+
-4/1| 1 |3/1--4/1| 2 |3/1-
| +---+ +---+ |
| |
|------------------------|
trunk probe results: 2 links
Link 1: u1 -- u2, num=1
1: 1/3/1 (P0) <---> 2/4/1 (P1)
Link 2: u1 -- u2, num=1
1: 1/4/1 (P1) <---> 2/3/1 (P0)
CPU to CPU packets are fine between 2 units.
Basically, you are looking that the CPU to CPU packets are okay all around the stack.
If all is great, "wr mem"
At any rate, after a renumber, you may have to change some VLAN memberships etc. Ultimately, though this should be readily repairable.
TAKE A BACKUP OF YOUR STACK CONFIG FIRST!!!
First I would do a "stack unconfigure me" or perhaps an "stack unconfigure all"... and a reload. Then I would do an "erase startup-config" followed by a reload.
If it says it is expecting spz10112, I would probably send that over and reload it to see if that message goes away. Then you can put back the bootrom of your choice, and it should go back through the upgrade procedure to upgrade the bootmonitor.
At any rate, it is a bit different being you are running the router firmware SPR08080B I think you said.
On a blank switch, we need to put an IP on an interface that is within the same subnet as your TFTP server or if that TFTP server is on a different subnet, you need to also put a next hop to point to your LAN's default-gateway.
Basically choose an interface like Ethernet 1/1/48 or whatever and put an IP on it within your network on a Layer-3 Interface.
switch>
switch# conf t
switch(config)# int e 1/1/48
switch(config-if-e1000-1/1/48)# ip add 10.1.2.3/24
switch(config-if-e1000-1/1/48)# enable
Not needed it the TFTP server is on the SAME LAN... /24 above is for the subnet mask 255.255.255.0 (adjust accordingly to your LAN subnet)
switch(config-if-e1000-1/1/48)# exit
switch(config)# ip route 0.0.0.0/0 10.1.2.1
To Downgrade to 10.1.12 boot monitor that it is expecting:
switch# copy tftp flash 10.1.2.4 spz10112.bin bootrom
Please do a "show flash" to verify it is present before reloading. Sometimes it takes a minute after the copy completes before it is really ready!
****************
Once everything is looking great, you can re-add the device to your stack.
Basically, you would put it back in and reconnect the cables. It is best if it has a matching bootrom and firmware!
Once you connect the stacking cables, if you do not get messages on the terminal that they changed state to UP, you would need to adjust the speed. I do not think this was an issue on the 7450's I dealt with though.
At this point
The entire stack should probably run
stack (config)# hitless-failover enable
Switch you add:
switch(config)# stack enable
Please do this in a maintenance window. If you have hitless-failover, only the new device should reboot. Otherwise the entire stack will likely reboot.
Next, you probably want the MAC addresses of each switch.... top to bottom; since, you want them numbered in order...
Do a "sh stack"
If all is well you will see a ring topology shown.
You are also looking for: "Standby u2 - protocols ready, can failover or manually switch over"
You really do not want to make any changes until everything comes into convergence within the stack. At that point, I would do a "wr mem"
Next to renumber, rerun your "stack secure-setup"
If you have your list of MAC addresses, it can walk you through renumbering all the switches.
Once complete, you want to validate communications
SSH@SWITCH#sh stack connection
Probing the topology. Please wait ...
SSH@SWITCH#
active standby
+---+ +---+
-4/1| 1 |3/1--4/1| 2 |3/1-
| +---+ +---+ |
| |
|------------------------|
trunk probe results: 2 links
Link 1: u1 -- u2, num=1
1: 1/3/1 (P0) <---> 2/4/1 (P1)
Link 2: u1 -- u2, num=1
1: 1/4/1 (P1) <---> 2/3/1 (P0)
CPU to CPU packets are fine between 2 units.
Basically, you are looking that the CPU to CPU packets are okay all around the stack.
If all is great, "wr mem"
At any rate, after a renumber, you may have to change some VLAN memberships etc. Ultimately, though this should be readily repairable.
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-03-2019 11:54 AM
Sorry, I forgot to post this update yesterday:
I have no idea how it happened but when I went to console into the problem switch earlier today, I didn't get the boot-mode prompt, I got the new switch prompt (empty running config). I ran the "show version" and both the primary and secondary flashes had the correct firmware. I guess whatever I did on Friday with the firmware and the bootrom took a while to process.
I had backed up the running-config last week and I managed to download it into the starting-config (initially there was no starting-config.txt file to download into until I made some small change to the empty running-config and wrote it to memory) but after a reload, I still had an empty running-config. Maybe the starting-config for the stack wasn't going to work as the starting-config for the switch.
I have no idea how it happened but when I went to console into the problem switch earlier today, I didn't get the boot-mode prompt, I got the new switch prompt (empty running config). I ran the "show version" and both the primary and secondary flashes had the correct firmware. I guess whatever I did on Friday with the firmware and the bootrom took a while to process.
I had backed up the running-config last week and I managed to download it into the starting-config (initially there was no starting-config.txt file to download into until I made some small change to the empty running-config and wrote it to memory) but after a reload, I still had an empty running-config. Maybe the starting-config for the stack wasn't going to work as the starting-config for the switch.
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-04-2019 12:40 PM
I think we're going to RMA the problem switch. We went through the procedures for having the switch "discovered" by the stack and it never worked. It seems to be that whatever the zero-input process or the spx configuration save in the switch's flash memory that configures it as a Port Extension is lost every time it reboots.
Thanks everyone for all your input.
Thanks everyone for all your input.
