topic Re: T710 and T610 running 6.1.1 firmware in Access Points - Indoor and Outdoor

T710 and T610 running 6.1.1 firmware

mpaine — Wed, 13 Mar 2024 19:21:10 GMT

I have a bunch of Ruckus T710 and T610 access points that are currently running 5.2.2.0.2069 firmware that I'd like to upgrade to 6.1.1.0.1274 firmware so that they are aligned with our Data Plane. When I trigger an upgrade on our controller for theses APs, the AP seems to complete the firmware upgrade, but then goes into what seems like a boot loop. It will just continually say "[system recovery for target failure]" after the upgrade completes. If I SSH into the AP, I can see that it is running 6.1.1.0.1274 firmware.

These AP models are said to be compatible with 6.1.1 per the Ruckus upgrade guide. Is there a common reason they would not be successfully upgrading?

Re: T710 and T610 running 6.1.1 firmware

Dilshad_Zafar — Mon, 18 Mar 2024 00:27:20 GMT

Hi @mpaine ,

If you are running 6.1.1 firmware, the recommended AP firmware is AP patch 6.1.1.0.1322 as there are similar known issues related to reboot due to target assert which are fixed in this patch :-

Regards,

Re: T710 and T610 running 6.1.1 firmware

mpaine — Mon, 18 Mar 2024 13:36:50 GMT

Re: T710 and T610 running 6.1.1 firmware

mpaine — Mon, 18 Mar 2024 13:36:18 GMT

@Dilshad_Zafar - this didn't seem to work. I upgraded a T710S to 6.1.1.0.1322 instead of 6.1.1.0.1274 and I still get the same boot looping behavior with the error message "AP was rebooted by the system because of [system recovery for target failure]."

Re: T710 and T610 running 6.1.1 firmware

Dilshad_Zafar — Mon, 18 Mar 2024 16:38:25 GMT

Hi @mpaine , how often is the reboot ? Can you check the uptime of the AP either from the vSZ GUI > AP menu or from the AP CLI "get uptime"?

Regards,

Re: T710 and T610 running 6.1.1 firmware

mpaine — Mon, 18 Mar 2024 16:43:37 GMT

@Dilshad_Zafar about every 5 minutes or so, see the attached event log

Re: T710 and T610 running 6.1.1 firmware

Dilshad_Zafar — Mon, 18 Mar 2024 17:04:38 GMT

Hi @mpaine , Could you go to the Zone advanced settings and check what is the AP reboot timeout configured there? May I also know how many APs are there in that zone and how many of them are showing this behaviour?

Regards,

Re: T710 and T610 running 6.1.1 firmware

mpaine — Mon, 18 Mar 2024 18:11:29 GMT

In the zone I am migrating from (5.2.2) the settings are 30 minutes and 2 hours as shown above. In the zone I am migrating to (6.1.1), the settings are also 30 minutes and 2 hours. There are about 30 T710 and T610 units in this zone that exhibit this behavior when upgraded.

Re: T710 and T610 running 6.1.1 firmware

Dilshad_Zafar — Mon, 18 Mar 2024 18:29:45 GMT

Thanks @mpaine , Could you please pull out AP support log from one of such APs and share the logs just beneath the string "### Kernel Oops ###"

Regards,

Re: T710 and T610 running 6.1.1 firmware

mpaine — Mon, 18 Mar 2024 18:31:47 GMT

This is what is displayed below:

### Kernel Oops ###
0xdf400000[0]: Image: type=1 index=1 total_boot=13

Re: T710 and T610 running 6.1.1 firmware

Dilshad_Zafar — Mon, 18 Mar 2024 18:38:49 GMT

Thanks, could you please paste the next 25-30 lines after this.

Regards,

Re: T710 and T610 running 6.1.1 firmware

mpaine — Tue, 19 Mar 2024 02:00:05 GMT

@Dilshad_Zafar

Apologies, here is the rest of that section and the next section:

### Kernel Oops ###
0xdf400000[0]: Image: type=1 index=1 total_boot=13

### Processes: ###
PID PPID USER VSZ STAT COMMAND
1 0 root 1156 S init
2 0 root 0 SW [kthreadd]
3 2 root 0 SW [ksoftirqd/0]
4 2 root 0 SW [kworker/0:0]
5 2 root 0 SW< [kworker/0:0H]
6 2 root 0 SW [kworker/u4:0]
7 2 root 0 SW [rcu_sched]
8 2 root 0 SW [rcu_bh]
9 2 root 0 SW [migration/0]
10 2 root 0 SW [watchdog/0]
11 2 root 0 SW [watchdog/1]
12 2 root 0 SW [migration/1]
13 2 root 0 SW [ksoftirqd/1]
14 2 root 0 SW [kworker/1:0]
15 2 root 0 SW< [kworker/1:0H]
16 2 root 0 SW< [khelper]
17 2 root 0 SW [irq/202-msmdata]
18 2 root 0 SW< [writeback]
19 2 root 0 SW< [bioset]
20 2 root 0 SW< [crypto]
21 2 root 0 SW< [kblockd]
22 2 root 0 SW [khubd]
23 2 root 0 SW [kswapd0]
24 2 root 0 SW [kworker/1:1]
25 2 root 0 SW [fsnotify_mark]
26 2 root 0 SW [kworker/u4:1]
33 2 root 0 DW [kworker/0:1]
34 2 root 0 SW [spi32766]
45 2 root 0 SW< [ipv6_addrconf]
73 2 root 0 SW< [deferwq]
74 2 root 0 SW [kworker/u4:2]
75 2 root 0 SW [ubi_bgt0d]
76 2 root 0 DW [kworker/1:2]
86 1 root 676 S < /sbin/udevd
108 1 root 7908 S < /usr/sbin/rsmd start
110 1 root 7880 S < watchdog
121 2 root 0 SW [ubi_bgt1d]
135 2 root 0 SW [ubi_bgt2d]
145 2 root 0 SW [ubifs_bgt2_0]
323 108 root 7884 S < /usr/sbin/timer start
330 1 root 736 S < /usr/sbin/rngd -b -o /dev/random -r /dev/hw_ran
342 2 root 0 SW< [gmac_workqueue]
353 2 root 0 SW< [nss_data_plane_]
354 2 root 0 SW< [nss_freq_queue]
355 2 root 0 SW< [coredump_wait]
449 1 root 8852 S < /usr/sbin/tcsd
513 2 root 0 SW< [alloc_task_wque]
515 2 root 0 SW< [alloc_task_wque]
519 2 root 0 SW< [alloc_task_wque]
622 2 root 0 SW [kworker/0:2]
636 1 root 1412 S < syslogd -C 256 -L -j 3 -f -1 -F 0 -y 7 -B kern@
641 1 root 8276 S < eved
720 1 root 1140 S < klogd -c 4
761 2 root 0 SW [kworker/0:3]
763 1 root 16888 S /usr/bin/stainfod
770 1 root 3100 S /usr/bin/mosquitto -c /tmp/mosquitto.conf
773 1 root 3308 S /usr/bin/matrix
783 1 root 7904 S < rksmcast
784 2 root 0 SW [kworker/1:3]
788 1 root 764 S < gpsp /etc/gpsconfig-csr.conf
798 1 root 1040 S < memcached -b -p /var/run/memcached.pid -s 127.0
801 1 root 21344 S < MsgDist ap_md -syslog -nonblocking
804 1 root 12144 S mdproxy -syslog -s udp 1812 1812 scg_radius -s
807 1 root 8664 S < sessionMgr -s
811 1 root 8292 S < /usr/sbin/dropbear -e /usr/bin/-rkscli -I 900 -
820 1 root 8620 S < dbclient -N -f -i /tmp/dataKey -L 8082:localhos
836 1 root 7352 S < hostapd -B -g/var/run/hostapd-global -P/var/run
840 1 root 8072 S < /usr/sbin/authd -p /var/run/authd.pid
852 1 root 7960 S < /usr/sbin/ftroamd -p /var/run/ftroamd.pid -b
859 1 root 8040 S < /usr/sbin/acsd -i wifi0
860 1 root 8096 S < /usr/sbin/acsd -i wifi1
862 1 root 8300 S < /usr/sbin/nbrd
898 1 root 4024 S < lighttpd -f /tmp/lighttpd.conf
932 1 root 9844 R N webs -s -p 18080 -P 8443 -b localhost
935 1 root 7888 S < /usr/sbin/portald
937 108 root 7980 S < /usr/sbin/rfwd start
948 1 root 8016 S < lldpd -C 0c:f4:d5:2e:92:70 -S Ruckus T710S Mult
952 948 root 8028 S < lldpd -C 0c:f4:d5:2e:92:70 -S Ruckus T710S Mult
955 1 root 8016 S < /usr/sbin/rccd
977 1 root 9920 S < /usr/sbin/snmpd udp:161,udp6:161 -p /var/run/sn
980 1 root 8124 S < tunnelmgr
983 1 root 7948 S < softgremgr
997 1 root 8084 S < /usr/sbin/mdnsfence -p /tmp/mdnsfence.pid -b
1000 1 root 7960 S < dpconfmgr
1023 1 root 8392 S < /usr/sbin/avpd -D
1039 1 root 11108 S < /usr/sbin/statd -D
1045 1 root 13752 S < /usr/sbin/cubic
1139 1 root 7876 S < hs20d
1146 1 root 8444 S < ztmd -s
1148 1 root 8388 S < /usr/sbin/clbd -S
1159 1 root 7916 S < /usr/sbin/channelfly -i wifi0 -q
1173 1 root 8192 S < /usr/sbin/channelfly -i wifi1 -q
1193 1 root 9924 S < /usr/sbin/wipingd
1195 1 root 7928 S < /usr/sbin/scand -S
1200 1 root 8128 S < idm -b -p /var/run/idm.pid
1206 1 root 8304 S N rogued -b -p /var/run/rogue_pid
1211 1 root 9084 S < chanbkgndscan -b -p /var/run/chanbkgndscan.pid
1216 1 root 9228 S < chanflybg -b -p /var/run/chanflybg.pid
1219 1 root 9820 S < cpd
1242 1 root 35928 S < /usr/sbin/qm_dpi -D -n 1
1249 1 root 7936 S < /usr/sbin/ipsecmgr
1253 1 root 7884 S < dbdc
1349 1 root 7880 S < rflow_radiod -b -m 2
1356 1 root 648 S < remoteclid -p 8083 -d 127.0.0.1
1370 1 root 9080 S /usr/bin/-rkscli
1394 1 root 22148 S < collectd -C /var/run/collectd.conf
1451 1356 root 1148 S < sh -c rkscli -c 'support;support show' 2>&1
1452 1451 root 9612 S < rkscli -c support;support show
2242 1452 root 1148 S < sh -c ps -e
2243 2242 root 1148 R < ps -e

###

As well, here is the same output from a different AP:

### Kernel Oops ###
0xdf400000[0]: Image: type=1 index=1 total_boot=43

### Processes: ###
PID PPID USER VSZ STAT COMMAND
1 0 root 1156 S init
2 0 root 0 SW [kthreadd]
3 2 root 0 SW [ksoftirqd/0]
5 2 root 0 SW< [kworker/0:0H]
7 2 root 0 RW [rcu_sched]
8 2 root 0 SW [rcu_bh]
9 2 root 0 SW [migration/0]
10 2 root 0 SW [watchdog/0]
11 2 root 0 SW [watchdog/1]
12 2 root 0 SW [migration/1]
13 2 root 0 SW [ksoftirqd/1]
15 2 root 0 SW< [kworker/1:0H]
16 2 root 0 SW< [khelper]
17 2 root 0 SW [irq/202-msmdata]
18 2 root 0 SW< [writeback]
19 2 root 0 SW< [bioset]
20 2 root 0 SW< [crypto]
21 2 root 0 SW< [kblockd]
22 2 root 0 SW [khubd]
23 2 root 0 SW [kswapd0]
24 2 root 0 SW [kworker/1:1]
25 2 root 0 SW [fsnotify_mark]
26 2 root 0 SW [kworker/u4:1]
33 2 root 0 SW [kworker/0:1]
34 2 root 0 SW [spi32766]
45 2 root 0 SW< [ipv6_addrconf]
73 2 root 0 SW< [deferwq]
74 2 root 0 SW [ubi_bgt0d]
82 2 root 0 SW [kworker/0:2]
86 1 root 708 S < /sbin/udevd
108 1 root 7908 S < /usr/sbin/rsmd start
110 1 root 7880 S < watchdog
119 2 root 0 SW [ubi_bgt1d]
134 2 root 0 SW [ubi_bgt2d]
145 2 root 0 SW [ubifs_bgt2_0]
367 108 root 7884 S < /usr/sbin/timer start
375 1 root 736 S < /usr/sbin/rngd -b -o /dev/random -r /dev/hw_ran
387 2 root 0 SW< [gmac_workqueue]
400 2 root 0 SW< [nss_data_plane_]
402 2 root 0 SW< [nss_freq_queue]
403 2 root 0 SW< [coredump_wait]
410 2 root 0 SW [kworker/u4:2]
496 1 root 10904 S < /usr/sbin/tcsd
560 2 root 0 SW< [alloc_task_wque]
564 2 root 0 SW< [alloc_task_wque]
566 2 root 0 SW< [alloc_task_wque]
685 1 root 1412 S < syslogd -C 256 -L -j 3 -f -1 -F 0 -y 7 -B kern@
690 1 root 8280 S < eved
769 1 root 1140 S < klogd -c 4
811 1 root 16888 S /usr/bin/stainfod
820 1 root 3296 S /usr/bin/mosquitto -c /tmp/mosquitto.conf
824 1 root 3724 S /usr/bin/matrix
834 1 root 7904 S < rksmcast
838 1 root 764 S < gpsp /etc/gpsconfig-csr.conf
848 1 root 1040 S < memcached -b -p /var/run/memcached.pid -s 127.0
851 1 root 21348 S < MsgDist ap_md -syslog -nonblocking
854 1 root 12144 S mdproxy -syslog -s udp 1812 1812 scg_radius -s
857 1 root 8672 S < sessionMgr -s
861 1 root 8292 S < /usr/sbin/dropbear -e /usr/bin/-rkscli -I 900 -
868 1 root 8864 S < dbclient -N -f -i /tmp/dataKey -L 8082:localhos
886 1 root 7356 S < hostapd -B -g/var/run/hostapd-global -P/var/run
890 1 root 8072 S < /usr/sbin/authd -p /var/run/authd.pid
902 1 root 8016 S < /usr/sbin/ftroamd -p /var/run/ftroamd.pid -b
905 1 root 8156 S < /usr/sbin/acsd -i wifi1
906 1 root 8156 S < /usr/sbin/acsd -i wifi0
907 1 root 8384 S < /usr/sbin/nbrd
944 1 root 4624 S < lighttpd -f /tmp/lighttpd.conf
981 1 root 7888 S < /usr/sbin/portald
983 108 root 8000 S < /usr/sbin/rfwd start
994 1 root 8016 S < lldpd -C 0c:f4:d5:2e:82:30 -S Ruckus T710S Mult
998 994 root 8024 S < lldpd -C 0c:f4:d5:2e:82:30 -S Ruckus T710S Mult
1001 1 root 8016 S < /usr/sbin/rccd
1023 1 root 10016 S < /usr/sbin/snmpd udp:161,udp6:161 -p /var/run/sn
1026 1 root 8124 S < tunnelmgr
1029 1 root 7948 S < softgremgr
1043 1 root 8088 S < /usr/sbin/mdnsfence -p /tmp/mdnsfence.pid -b
1046 1 root 8096 S < dpconfmgr
1067 1 root 8392 S < /usr/sbin/avpd -D
1083 1 root 11112 S < /usr/sbin/statd -D
1089 1 root 15004 S < /usr/sbin/cubic
1169 1 root 7876 S < hs20d
1175 1 root 8444 S < ztmd -s
1178 1 root 8388 S < /usr/sbin/clbd -S
1189 1 root 7968 S < /usr/sbin/channelfly -i wifi0 -q
1203 1 root 8192 S < /usr/sbin/channelfly -i wifi1 -q
1223 1 root 7928 S < /usr/sbin/scand -S
1224 1 root 9924 S < /usr/sbin/wipingd
1230 1 root 8128 S < idm -b -p /var/run/idm.pid
1238 1 root 8360 S N rogued -b -p /var/run/rogue_pid
1245 1 root 9168 S < chanbkgndscan -b -p /var/run/chanbkgndscan.pid
1250 1 root 9184 S < chanflybg -b -p /var/run/chanflybg.pid
1253 1 root 9820 S < cpd
1283 1 root 7936 S < /usr/sbin/ipsecmgr
1287 1 root 7884 S < dbdc
1382 1 root 7880 S < rflow_radiod -b -m 2
1389 1 root 648 S < remoteclid -p 8083 -d 127.0.0.1
1404 1 root 9080 S /usr/bin/-rkscli
1445 1 root 21684 S < collectd -C /var/run/collectd.conf
1650 1 root 35928 S < /usr/sbin/qm_dpi -D -n 1
19859 2 root 0 SW [kworker/1:0]
26164 1389 root 1148 S < sh -c rkscli -c 'support;support show' 2>&1
26165 26164 root 9612 S < rkscli -c support;support show
26264 2 root 0 SW [kworker/1:2]
26783 26165 root 1148 S < sh -c ps -e
26784 26783 root 1148 R < ps -e

###

Re: T710 and T610 running 6.1.1 firmware

mpaine — Thu, 21 Mar 2024 13:22:15 GMT

It appears to correct this issue, the T710 and T610 models must be upgraded to 6.1.1 and then immediately factory reset and re-provisioned. Once this procedure is done, the rebooting issue stops and the AP functions as normal on 6.1.1 firmware.