ICX7150 firmware upgrade, Job timed out after 120 mins, please retry later.

[dennis_urbanczy]

Hello,

I have a site with several ICX7150 switches. A few 48P, a few 24P and one C12P. They were previously running on SPS08095k and I wanted to upgrade to SPS08095r. I can currently “only” manage them via vSZ or remotely via SSH using VPN.

The network is simple: no LAGs, no stacks, nothing special, a simple tree structure, just a few VLANs. All the switches are new and were only installed two or three months ago.

So I initiated the upgrade for the entire switch group. This worked for 11 of the 13 switches on this site, but not for one 48P and the C12P. The status is “Failed” and under Failure Reason under Firmware History it says “Job timed out after 120 mins, please retry later.”

I've repeated the upgrade process for these two switches after a warm reboot but it didn't change. It takes like 20 to 25 min and the Failure Reason is still the same. While the C12P is at

I also tried HTTPS and uploaded the SPS08095rufi.bin to a webspace and tried the following:

SSH@icx7150#copy https flash A.B.C.D SPS08095rufi.bin primary port 443
SSH@icx7150#Feb 8 14:44:42:HTTP Request Failed for Request ID: 1. HTTP Connection Error...

This also does not work with the switches that have already been updated. So I think there might be another problem.

SCP and TFTP are out, as I am currently unable to set up a corresponding server.

What should I try next or could be the issue?

[BenBeck]

ICX7150 flash is quite slow compared to the rest of our lineup. You could try adding this and giving it another shot:

device(config)# flash-timeout 120

device(config)# wr mem

If it still fails in SZ, then you may need to do it manually. You can issue a 'show cli-command-history' and you should be able to see the 'copy https flash' command that SZ is using. You can run that manually from the CLI and see if the copy completes that way. Then you would simply need to wr mem and boot into the new code. If none of that works, please open a case and we can assist. 

[dennis_urbanczy]

I tried adding # flash-timeout 120 but ended up the same way.

It also seems to be the case that this is repeatedly added by SZ itself:

SSH@icx7150(config)#show cli-command-history wide | include timeout
15 telnet_1 Un-authenticated user 192.168.100.249 Jul 21 00:04:54 flash-timeout 120
34 telnet_1 Un-authenticated user 192.168.100.249 Jul 20 23:59:17 flash-timeout 120
53 telnet_1 Un-authenticated user 192.168.100.249 Jul 20 23:53:39 flash-timeout 120
60 ssh_7 admin 192.168.100.245 Jul 20 23:50:52 flash-timeout 120
83 telnet_1 Un-authenticated user 192.168.100.249 Jul 20 06:05:50 flash-timeout 120
107 telnet_1 Un-authenticated user 192.168.100.249 Jul 20 06:00:14 flash-timeout 120
126 telnet_1 Un-authenticated user 192.168.100.249 Jul 20 05:54:38 flash-timeout 120
148 telnet_1 Un-authenticated user 192.168.100.249 Jul 20 04:56:42 flash-timeout 120
167 telnet_1 Un-authenticated user 192.168.100.249 Jul 20 04:54:15 flash-timeout 120

Slno 60 was my attempt.

Then I checked for previous SZ attemps to upgrade the flash and found no executed commands 'copy https flash'. There simply aren't any, even after a manual push by me via the SZ, even going back to when the switches were initially configured.

I may even have found a clue as to why: I also cannot start a CLI session for these two switches via the SZ. It seems the SZ cannot properly communicate with the devices. I also cannot configure any ports, for example. It appears that the SZ's communication with the devices is disrupted on a larger scale. However, they are shown online and I can reboot them via SZ

I checked the 'Event' Tab of both the switches and these three 'critical' error messages appear in the 'event' tabs of both switches.

'Application nginx failed recovery and functionality provided by it will not be available until failure reason is remedied (may require manual intervention)'

'Application uwsgi failed recovery and functionality provided by it will not be available until failure reason is remedied (may require manual intervention)'

'Application PySzAgtSrv.py failed recovery and functionality provided by it will not be available until failure reason is remedied (may require manual intervention'

[Mayank]

Hi dennis_urbanczy ,

Thank you for posting your query !!!

I understand that your trying to upgrade the firmware but getting time out issue.

Could you please try again Via USB Or TFTP Server as shown below :

TFTP:

1. Download the software on a TFTP server, the TFTP server should have ping connectivity with the switch.

2. Get console access and test the ping connectivity with the command brocade# ping x.x.x.x. Note: if you have a stack get console access from the active controller.

3. Issue the Show flash command to confirm you have the same version on both partitions (primary and secondary partition)

NOTE: ON the release notes and zip file downloaded you can find the required boot code for the desired image version.

4. Upgrade the image software version on primary/secondary partition

Switch# copy tftp flash x.x.x.x <filename> primary/secondary ("show flash" to confirm)

NOTE: On a stack environment implement the upgrade from the active controller and it will copied to the member automatically.

5. Boot from primary/secondary partition

Switch# boot system flash primary/secondary.

USB:

To Copy The Files :
Copy Disk0 Flash < File Name > < Location > [ Here We Had Used Bootrom And Secondary ]
Copy Flash Flash Primary [ To Copy The Code From Secondary To Primary Sector ]
To Check :
Show Version
Show Files Disk0
Show Flash
To Boot Up :
Boot System Flash Secondary [ As We Had Loaded The Code Here ]

Moving Forward If this issue is not resolved , Please log a ticket with the below link so that we will help you further.

https://support.ruckuswireless.com/contact-us

I hope this information helps you

Please feel free to leave us a message if any concerns

Note: Please feel free to mark the post as ACCEPTED SOLUTIONS if its addressed your query.

Thanks

[dennis_urbanczy]

Unfortunately, I am currently unable to set up a publicly accessible TFTP server. The installation location is also 3 hours away from me. Of course, an on-site check would probably be best, but as long as it is possible remotely, I would like to avoid it.