08-24-2022 12:41 AM
Hi Ruckus Community,
We have 800++ ICX 7150 switches spread across many locations (branch office) connected to SmartZone in Data Center via Cisco SD-WAN. Almost 30% of the switch are indicated offline in the SmartZone, but actually, they are online and working properly. All the switches which disconnected/offline in the SmartZone are stuck at Query state.
Our switches are :
- all using UFI images
- firmware is vary , some using 08090d some using 08095d
- ports ranging from 24 to 48 port
- the WAN latency between DC and branch is under 100ms
Troubleshooting that we try:
- trigger the 'query registrar'
- disable and enable the manager on the switch side
- doing zeroize on crypto key and re-regenerate the SSL certificate on the switch, the doing restart on the switch
- not all the switch is offline, even on the same branch we often found that only several switches is indicated offline on the SmartZone
After we do the step above, it appears some switch is back to an online state again, but after several days the switch is back to offline again
the last step we do is restarting all offline switches but this appears doesn't seem to help that much
is there any WAN link or firewall policy consideration or limitation that causing those issues?
08-24-2022 09:53 PM
Heres the output below ,
ICX SWITCH
1.show version
Copyright (c) Ruckus Networks, Inc. All rights reserved.
UNIT 1: compiled on Sep 23 2019 at 00:00:25 labeled as SPR08090d
(32539748 bytes) from Secondary SPR08090d.bin (UFI)
SW: Version 08.0.90dT213
Compressed Secondary Boot Code size = 786944, Version:10.1.15T225 (mnz10115)
Compiled on Thu Jan 31 07:08:55 2019
HW: Stackable ICX7150-48-POE
==========================================================================
UNIT 1: SL 1: ICX7150-48P-4X1G POE 48-port Management Module
Serial #:FEC3241R015
Software Package: ICX7150_BASE_L3_SOFT_PACKAGE
Current License: 4X1G
P-ASIC 0: type B160, rev 11 Chip BCM56160_B0
==========================================================================
UNIT 1: SL 2: ICX7150-2X1GC 2-port 2G Module
==========================================================================
UNIT 1: SL 3: ICX7150-4X10GF 4-port 40G Module
==========================================================================
1000 MHz ARM processor ARMv7 88 MHz bus
8192 KB boot flash memory
2048 MB code flash memory
1024 MB DRAM
STACKID 1 system uptime is 61 day(s) 13 hour(s) 18 minute(s) 20 second(s)
The system started at 16:23:12 GMT+00 Fri Jun 24 2022
The system : started=warm start reloaded=by "reload"
2.show sz status
============ SZ Agent State Info ===================
Config Status: None Operation Status: Enabled
State: SZ QUERY Prev State: SZ SSH CONNECTED Event: SZ QUERY RESPONSE
SWR List : None
Active List : 172.15.98.73, 172.15.98.29
DHCP Option 43 : No
DHCP Opt 43 List : None
Passive List : None
Merged List : 172.15.98.73, 172.15.98.29
Merged Idx: 0 IP : 172.15.98.73
Switch registrar host: sw-registrar.ruckuswireless.com
Switch registrar discovery retry count: 14
Switch registrar host resolve failure count: 14
SZ IP Used : 172.15.98.29
SZ Query Status :
In Progress. Response Not Received.
SSH Tunnel Status - :
Tunnel Status : Not Initiated
CLI IP/Port : /0
SNMP IP/Port : /0
Syslog IP/Port : /0
HTTP SERVER IP/Port: /0
HTTP CLIENT IP/Port: /0
Timer Status : Running
Timer Details in msec
Start Time Stamp : 962187078
Duration : 64601212
Current Time Stamp: 1023435120
Time Left : 3353170
3.sh sz client-log detail
there is no such command in the switch , instead I will display you the 'sh sz logs' command
Aug 24 11:00:56:https_connmgr_send_request>Entered.
Aug 24 11:00:56:sz_execute_state_machine>Exit with state/event: SZ QUERY/5, TIMER/2002 RC: 1
Aug 24 11:00:57:sz_execute_state_machine>Entering with state/event: SZ QUERY/5, SZ QUERY RESPONSE/2007
Aug 24 11:00:57:sz_parse_sz_query_response -- Status: 600 <<
Aug 24 11:00:57:sz_execute_state_machine>Exit with state/event: SZ QUERY/5, SZ QUERY RESPONSE/2007 RC: 1
Aug 24 11:00:57:HTTP Request Error:Http remote connection close called.
Aug 25 03:48:38:Sending SZ_SWR_QUERY_RESTART event...
Aug 25 03:48:38:Failed to resolve sw-registrar.ruckuswireless.com
Aug 25 03:49:38:Failed to resolve sw-registrar.ruckuswireless.com
Aug 25 03:50:38:Failed to resolve sw-registrar.ruckuswireless.com
End i/max/iter 167/167/0
4.show sz server-log detail
there is no such command in the switch
SMARTZONE CONTROLLER
1.vSZ#show service
2.vSZ#show version
3.vSZ#show system-capacity
4.vSZ#show cpuinfo
5.vSZ#show diskinfo
6.vSZ#show meminfo
Regards ,
Fauzan
08-24-2022 09:55 PM
Heres the output below ,
ICX SWITCH
1.show version
Copyright (c) Ruckus Networks, Inc. All rights reserved.
UNIT 1: compiled on Sep 23 2019 at 00:00:25 labeled as SPR08090d
(32539748 bytes) from Secondary SPR08090d.bin (UFI)
SW: Version 08.0.90dT213
Compressed Secondary Boot Code size = 786944, Version:10.1.15T225 (mnz10115)
Compiled on Thu Jan 31 07:08:55 2019
HW: Stackable ICX7150-48-POE
==========================================================================
UNIT 1: SL 1: ICX7150-48P-4X1G POE 48-port Management Module
Serial #:FEC3241R015
Software Package: ICX7150_BASE_L3_SOFT_PACKAGE
Current License: 4X1G
P-ASIC 0: type B160, rev 11 Chip BCM56160_B0
==========================================================================
UNIT 1: SL 2: ICX7150-2X1GC 2-port 2G Module
==========================================================================
UNIT 1: SL 3: ICX7150-4X10GF 4-port 40G Module
==========================================================================
1000 MHz ARM processor ARMv7 88 MHz bus
8192 KB boot flash memory
2048 MB code flash memory
1024 MB DRAM
STACKID 1 system uptime is 61 day(s) 13 hour(s) 18 minute(s) 20 second(s)
The system started at 16:23:12 GMT+00 Fri Jun 24 2022
The system : started=warm start reloaded=by "reload"
2.show sz status
============ SZ Agent State Info ===================
Config Status: None Operation Status: Enabled
State: SZ QUERY Prev State: SZ SSH CONNECTED Event: SZ QUERY RESPONSE
SWR List : None
Active List : 172.15.98.73, 172.15.98.29
DHCP Option 43 : No
DHCP Opt 43 List : None
Passive List : None
Merged List : 172.15.98.73, 172.15.98.29
Merged Idx: 0 IP : 172.15.98.73
Switch registrar host: sw-registrar.ruckuswireless.com
Switch registrar discovery retry count: 14
Switch registrar host resolve failure count: 14
SZ IP Used : 172.15.98.29
SZ Query Status :
In Progress. Response Not Received.
SSH Tunnel Status - :
Tunnel Status : Not Initiated
CLI IP/Port : /0
SNMP IP/Port : /0
Syslog IP/Port : /0
HTTP SERVER IP/Port: /0
HTTP CLIENT IP/Port: /0
Timer Status : Running
Timer Details in msec
Start Time Stamp : 962187078
Duration : 64601212
Current Time Stamp: 1023435120
Time Left : 3353170
3.sh sz client-log detail
there is no such command in the switch , instead I will display you the 'sh sz logs' command
Aug 24 11:00:56:https_connmgr_send_request>Entered.
Aug 24 11:00:56:sz_execute_state_machine>Exit with state/event: SZ QUERY/5, TIMER/2002 RC: 1
Aug 24 11:00:57:sz_execute_state_machine>Entering with state/event: SZ QUERY/5, SZ QUERY RESPONSE/2007
Aug 24 11:00:57:sz_parse_sz_query_response -- Status: 600 <<
Aug 24 11:00:57:sz_execute_state_machine>Exit with state/event: SZ QUERY/5, SZ QUERY RESPONSE/2007 RC: 1
Aug 24 11:00:57:HTTP Request Error:Http remote connection close called.
Aug 25 03:48:38:Sending SZ_SWR_QUERY_RESTART event...
Aug 25 03:48:38:Failed to resolve sw-registrar.ruckuswireless.com
Aug 25 03:49:38:Failed to resolve sw-registrar.ruckuswireless.com
Aug 25 03:50:38:Failed to resolve sw-registrar.ruckuswireless.com
End i/max/iter 167/167/0
4.show sz server-log detail
there is no such command in the switch
SMARTZONE CONTROLLER
1.vSZ#show service
2.vSZ#show version
3.vSZ#show system-capacity
4.vSZ#show cpuinfo
5.vSZ#show diskinfo
6.vSZ#show meminfo
Regards ,
Fauzan
08-30-2022 11:24 PM
I have the same problem, is there any solution that can help to solve it?
04-05-2023 07:12 AM
Hello, I would like to ask about a switch Mod. ICX7150-48-4X1G a Sys port lit up in orange and the send in green and no network port works even by turning the device off and on again