topic Re: Odd behaviour on network - RSTP and BDPU sanity check in ICX Switches

Odd behaviour on network - RSTP and BDPU sanity check

Killerbot — Sun, 27 Oct 2024 18:37:01 GMT

I need to preface this with I'm not a network specialist. I know a little from Network+ and CCNA many years ago but new to the world of Ruckus and, as I work in education, it's typically only something I dabble in when there's an issue.

We experienced some odd behaviour at one of our sites on Friday and I'm going out tomorrow as the troubleshooting needs a local console connection.

I'll detail the odd behaviour we're seeing, and my plans to troubleshoot on site, but was hoping someone could sanity check the example config from the core (cleaned to remove any potential security issues).

I see 802.1w is enabled but I don't see any ports configured as edge nor admin, no bdpu guard nor any 'loop protect'. Some of this could be assumed in Ruckus' 802.1w config but can't see anything to suggest that may be the case. CPU usage wasn't significantly high when I got them to check for me on site so it may not be a loop, but I've come across 2 'loop' issues in the last year with these Ruckus installs and that's 2 more than I've come across in over a decade which makes me question the setup.

I'll post an example core and edge setup below but wanted to detail the behaviour and see if anything jumps out to those of you who are kindly reading this.

SOME traffic could not access the internet. But there was no rhyme nor reason to it.

For example, device with ip 192.168.0.10/24 could ping it’s default gateway and access the internet. Device 192.168.0.11/24 could ping it’s default gateway but could NOT access the internet. .10 could ping the default route of the core, .11 couldn’t. The setup as is has been running for almost a year now and working fine until late Friday afternoon.

Annoyingly I can’t access the device that sits at the default route of the core as it’s our ISPs firewall but they’re assuring us that nothing has changed. The only changes I’ve been made aware of at the site are old handsets have been reprovisioned for another provider and this caused a similar issue at another site with Ruckus (I’m awaiting details on the actual issue but I get the impression they’re not actually sure). Again, this was done earlier in the week but I believe the VOIP providers are making changes remotely to troubleshoot various issues so wondering if they’ve done something that’s potentially affecting the network.

I’m going to head to site and do the usual show cpu, log, 802.1, dm raw, etc. to see if anything jumps out at me but would appreciate it if someone could give the setup below a once over and tell me if I need to be configuring edge ports and admin ports as well as enabling bdpu-guard or any other form of loop protect (and disabling said protection on uplinks).

I’ve removed a bunch of vlans and obfuscated anything remotely relating to actual addressing but hopefully it’s enough to tell you if we’re at risk of loops or broadcast storms currently.
Original got marked as spam so trying again with the configurations to follow.

Appreciate any advice people may have with regards to troubleshooting further. I’ll look at active traffic first but eventual plan if I can’t suss it is to all the lags down and disable all ports on the core, other than the link to the SRV stack (as that’s where the internet sits). See if things stabilise and then slowly bring interfaces back up until there’s an issue.

Re: Odd behaviour on network - RSTP and BDPU sanity check

Killerbot — Sun, 27 Oct 2024 18:37:53 GMT

Cleaned Core

ver 09.0.10h_cd2T243
!
stack unit 1
module 1 icx7550-24f-port-management-module
module 2 icx7550-100g-2port-200g-module
priority 128
stack-port ethernet 1/2/1
stack-port ethernet 1/2/2
stack unit 2
module 1 icx7550-24f-port-management-module
module 2 icx7550-100g-2port-200g-module
stack-port ethernet 2/2/1
stack-port ethernet 2/2/2
stack enable
stack mac OBFUSCATED
!
hitless-failover enable
!
!

!
!
global-stp
!
lag CabB_Link dynamic id 1
ports ethe 1/1/1 ethe 2/1/1
!
lag CabC_Link dynamic id 2
ports ethe 1/1/2 ethe 2/1/2
!
lag CabD0_Link dynamic id 3
ports ethe 1/1/3 ethe 2/1/3
!
lag CabE_Link dynamic id 4
ports ethe 1/1/4 ethe 2/1/4
!
lag CabG_Link dynamic id 5
ports ethe 1/1/5 ethe 2/1/5
!
lag CabH_Link dynamic id 6
ports ethe 1/1/6 ethe 2/1/6
!
lag SRV-WiFi_Link dynamic id 7
ports ethe 1/1/7 ethe 2/1/7
!
lag ESX01 static id 10
ports ethe 1/1/13 ethe 2/1/13
!
lag ESX02 static id 11
ports ethe 1/1/14 ethe 2/1/14
!
lag ESX03 static id 12
ports ethe 1/1/15 ethe 2/1/15
!
lag ESX04 static id 13
ports ethe 1/1/16 ethe 2/1/16
!
lag Link_to_SRV dynamic id 99
ports ethe 1/1/21 to 1/1/24 ethe 2/1/21 to 2/1/24
!
!
vlan 1 name DEFAULT-VLAN by port
spanning-tree priority 2048
!
vlan 2 name OBFUSCATED by port
tagged ethe 1/1/8 to 1/1/10 ethe 2/1/8 to 2/1/9 lag 7
untagged lag 1 to 6 lag 99
spanning-tree 802-1w
spanning-tree 802-1w hello-time 3 max-age 10 priority 2048
!
vlan 3 name OBFUSCATED by port
tagged lag 5 lag 99
untagged ethe 1/1/19 to 1/1/20 ethe 2/1/19 to 2/1/20 lag 10 to 13
spanning-tree 802-1w
spanning-tree 802-1w hello-time 3 max-age 10 priority 2048
!
vlan 5 name OBFUSCATED by port
tagged ethe 1/1/9 to 1/1/12 ethe 2/1/8 to 2/1/12 lag 1 to 6
spanning-tree 802-1w
spanning-tree 802-1w hello-time 3 max-age 10 priority 2048
!
vlan 7 name OBFUSCATED by port
tagged ethe 1/1/9 to 1/1/12 ethe 2/1/8 to 2/1/12 lag 1 to 6
spanning-tree 802-1w
spanning-tree 802-1w hello-time 3 max-age 10 priority 2048
!
vlan 11 name OBFUSCATED by port
tagged ethe 1/1/9 to 1/1/12 ethe 2/1/8 to 2/1/12 lag 1 to 6
spanning-tree 802-1w
spanning-tree 802-1w hello-time 3 max-age 10 priority 2048
!
vlan 12 name OBFUSCATED by port
tagged ethe 1/1/9 to 1/1/12 ethe 2/1/8 to 2/1/12 lag 1 to 6
untagged lag 7
spanning-tree 802-1w
spanning-tree 802-1w hello-time 3 max-age 10 priority 2048
!
vlan 13 name OBFUSCATED by port
tagged ethe 1/1/9 to 1/1/12 ethe 2/1/8 to 2/1/12 lag 1 to 6
spanning-tree 802-1w
spanning-tree 802-1w hello-time 3 max-age 10 priority 2048
!
vlan 16 name OBFUSCATED by port
tagged ethe 1/1/9 to 1/1/12 ethe 2/1/8 to 2/1/12 lag 1 to 6
spanning-tree 802-1w
spanning-tree 802-1w hello-time 3 max-age 10 priority 2048
!
vlan 22 name OBFUSCATED by port
tagged ethe 1/1/8 to 1/1/12 ethe 2/1/8 to 2/1/12 lag 1 to 6
spanning-tree 802-1w
spanning-tree 802-1w hello-time 3 max-age 10 priority 2048
!
vlan 23 name OBFUSCATED by port
tagged ethe 1/1/8 to 1/1/12 ethe 2/1/8 to 2/1/12 lag 1 to 6
spanning-tree 802-1w
spanning-tree 802-1w hello-time 3 max-age 10 priority 2048
!
vlan 27 name OBFUSCATED by port
tagged ethe 1/1/9 to 1/1/12 ethe 2/1/8 to 2/1/12 lag 1 to 6 lag 99
spanning-tree 802-1w
spanning-tree 802-1w hello-time 3 max-age 10 priority 2048
!
vlan 28 name OBFUSCATED by port
tagged ethe 1/1/9 to 1/1/12 ethe 2/1/8 to 2/1/12 lag 1 to 6
spanning-tree 802-1w
spanning-tree 802-1w hello-time 3 max-age 10 priority 2048
!
!
!
!
!
!
ip route 0.0.0.0/0 OBFUSCATED
!
!
hostname Core
ip dns server-address OBFUSCATED
ip tftp blocksize 8192
!
!
ntp
!
!
!
!
!
!
!
!
!
interface management 1
!
interface ve 2
ip address 192.168.129.1 255.255.255.128
ip helper-address OBFUSCATED
ip helper-address OBFUSCATED
!
interface ve 3
ip address 192.168.129.129 255.255.255.128
ip helper-address OBFUSCATED
ip helper-address OBFUSCATED
!
interface ve 5
ip address 192.168.130.1 255.255.255.0
ip helper-address OBFUSCATED
ip helper-address OBFUSCATED
!
interface ve 7
ip address 192.168.132.1 255.255.255.128
ip helper-address OBFUSCATED
ip helper-address OBFUSCATED
!
interface ve 11
ip address 192.168.134.1 255.255.255.128
ip helper-address OBFUSCATED
ip helper-address OBFUSCATED
!
interface ve 12
ip address 192.168.135.1 255.255.255.0
ip helper-address OBFUSCATED
ip helper-address OBFUSCATED
!
interface ve 13
ip address 192.168.136.1 255.255.255.128
ip helper-address OBFUSCATED
ip helper-address OBFUSCATED
!
interface ve 16
ip address 192.168.137.1 255.255.255.128
ip helper-address OBFUSCATED
ip helper-address OBFUSCATED
!
interface ve 22
ip address 192.168.139.65 255.255.255.192
ip helper-address OBFUSCATED
ip helper-address OBFUSCATED
!
interface ve 23
ip address 192.168.139.129 255.255.255.192
ip helper-address OBFUSCATED
ip helper-address OBFUSCATED
!
interface ve 27
ip address 192.168.140.1 255.255.255.192
ip helper-address OBFUSCATED
ip helper-address OBFUSCATED
!
interface ve 28
ip address 192.168.140.65 255.255.255.192
ip helper-address OBFUSCATED
ip helper-address OBFUSCATED
!
interface ve 31
ip address 192.168.141.1 255.255.255.128
ip helper-address OBFUSCATED
ip helper-address OBFUSCATED
!
interface ve 32
ip address 192.168.141.129 255.255.255.128
ip helper-address OBFUSCATED
ip helper-address OBFUSCATED
!
interface ve 33
ip address 192.168.142.1 255.255.255.128
ip helper-address OBFUSCATED
ip helper-address OBFUSCATED
!
interface ve 34
ip address 192.168.142.129 255.255.255.128
ip helper-address OBFUSCATED
ip helper-address OBFUSCATED
!
interface ve 35
ip address 192.168.143.1 255.255.255.128
ip helper-address OBFUSCATED
ip helper-address OBFUSCATED
!
interface ve 41
ip helper-address OBFUSCATED
ip helper-address OBFUSCATED
!
interface ve 51
ip address 192.168.146.1 255.255.254.0
ip helper-address OBFUSCATED
ip helper-address OBFUSCATED
!
interface ve 52
ip address 192.168.148.1 255.255.254.0
ip helper-address OBFUSCATED
ip helper-address OBFUSCATED
!
interface ve 53
ip address 192.168.150.1 255.255.254.0
ip helper-address OBFUSCATED
ip helper-address OBFUSCATED
!
interface ve 56
ip address 192.168.155.1 255.255.255.0
ip helper-address OBFUSCATED
ip helper-address OBFUSCATED
!
interface ve 58
ip address 192.168.156.129 255.255.255.128
ip helper-address OBFUSCATED
ip helper-address OBFUSCATED
!
interface ve 59
ip address 192.168.157.1 255.255.255.128
ip helper-address OBFUSCATED
ip helper-address OBFUSCATED
!
interface ve 60
ip address 192.168.158.1 255.255.255.0
ip helper-address OBFUSCATED
ip helper-address OBFUSCATED
!
interface ve 61
ip address 192.168.159.1 255.255.255.0
ip helper-address OBFUSCATED
ip helper-address OBFUSCATED
!
interface ve 156
ip address 192.168.156.1 255.255.255.128
ip helper-address OBFUSCATED
ip helper-address OBFUSCATED
!
!

!
!

!
!
!
username OBFUSCATED password 1 OBFUSCATED

!
aaa authentication login default local
!
aaa authentication web-server default local
!
aaa authentication snmp-server default local
!
ip dhcp-client disable
!
restconf enable
!

!
web-management disable
!

!
no telnet server
!
manager registrar
manager registrar-list device.eu.ruckus.cloud 34.89.230.65
manager port-list 987
!
cli timeout 60
!
banner motd ^C
This is a private system operated by .....
!

end

Re: Odd behaviour on network - RSTP and BDPU sanity check

Killerbot — Sun, 27 Oct 2024 18:38:26 GMT

Cleaned Edge

ver 10.0.10a_cd3T253
!
stack unit 1
module 1 icx8200-48-port-management-module
module 2 icx8200-4-sfp-plus-port-100g-module
priority 128
stack-port ethernet 1/2/1
stack-port ethernet 1/2/3
stack unit 2
module 1 icx8200-48-port-management-module
module 2 icx8200-4-sfp-plus-port-100g-module
stack-port ethernet 2/2/1
stack-port ethernet 2/2/3
stack enable
stack mac OBFUSCATED
!
hitless-failover enable
!
!

!
!
global-stp
!
lag Core_Link dynamic id 1
ports ethe 1/2/2 ethe 2/2/2
!
!
vlan 1 name DEFAULT-VLAN by port
spanning-tree
!
vlan 2 name OBFUSCATED by port
untagged ethe 1/1/1 ethe 1/1/3 lag 1
spanning-tree 802-1w
spanning-tree 802-1w hello-time 3 max-age 10 priority 16384
!
vlan 3 name OBFUSCATED by port
tagged lag 1
untagged ethe 1/1/9
spanning-tree 802-1w
spanning-tree 802-1w hello-time 3 max-age 10 priority 16384
!
vlan 5 name OBFUSCATED by port
tagged lag 1
untagged ethe 1/1/5 ethe 1/1/7 ethe 1/1/11 ethe 1/1/25 ethe 1/1/29 ethe 1/1/31 ethe 1/1/33 ethe 2/1/2 ethe 2/1/4 ethe 2/1/6 ethe 2/1/8
spanning-tree 802-1w
spanning-tree 802-1w hello-time 3 max-age 10 priority 16384
!
vlan 7 name OBFUSCATED by port
tagged lag 1
spanning-tree 802-1w
spanning-tree 802-1w hello-time 3 max-age 10 priority 16384
!
vlan 11 name OBFUSCATED by port
tagged lag 1
spanning-tree 802-1w
spanning-tree 802-1w hello-time 3 max-age 10 priority 16384
!
vlan 12 name OBFUSCATED by port
tagged lag 1
spanning-tree 802-1w
spanning-tree 802-1w hello-time 3 max-age 10 priority 16384
!
vlan 13 name OBFUSCATED by port
tagged lag 1
untagged ethe 1/1/13 ethe 1/1/15 ethe 1/1/17 ethe 1/1/19 ethe 1/1/21 ethe 1/1/23 ethe 1/1/27 ethe 2/1/10 ethe 2/1/12 ethe 2/1/14 ethe 2/1/16 ethe 2/1/18 ethe 2/1/20
spanning-tree 802-1w
spanning-tree 802-1w hello-time 3 max-age 10 priority 16384
!
vlan 16 name OBFUSCATED by port
tagged lag 1
untagged ethe 1/1/37 ethe 1/1/39
spanning-tree 802-1w
spanning-tree 802-1w hello-time 3 max-age 10 priority 16384
!
vlan 27 name OBFUSCATED by port
tagged lag 1
spanning-tree 802-1w
spanning-tree 802-1w hello-time 3 max-age 10 priority 16384
!
!
!
!
!
ip route 0.0.0.0/0 OBFUSCATED
!
!
hostname CabH
ip dns server-address OBFUSCATED
ip tftp blocksize 8192
!
!
!
!
!
!
!
interface management 1
!
interface ve 1
!
interface ve 2
ip address OBFUSCATED
!
!

!
!

!
!
!
username OBFUSCATED password 1 OBFUSCATED
!
aaa authentication login default local
!
aaa authentication web-server default local
!
aaa authentication snmp-server default local
!
ip dhcp-client ve default
!
restconf enable
!

!
manager registrar
manager registrar-list device.eu.ruckus.cloud 34.89.230.65
manager port-list 987
!
cli timeout 60
!
banner motd ^C
This is a private system operated by ...
!

end

Re: Odd behaviour on network - RSTP and BDPU sanity check

Killerbot — Sun, 27 Oct 2024 18:45:36 GMT

!
!

!
!
!
username OBFUSCATED password 1 OBFUSCATED

!
aaa authentication login default local
!
aaa authentication web-server default local
!
aaa authentication snmp-server default local
!
ip dhcp-client disable
!
restconf enable
!

!
web-management disable
!

!
no telnet server
!
manager registrar
manager registrar-list device.eu.ruckus.cloud 34.89.230.65
manager port-list 987
!
cli timeout 60
!
banner motd ^C
This is a private system operated by .....
!

end

Re: Odd behaviour on network - RSTP and BDPU sanity check

Killerbot — Sun, 27 Oct 2024 18:47:06 GMT

Doesn't seem to let me want to paste the core code. Perhaps too long.

If there are particular areas you'd highlight please let me know and I'll post them.

Re: Odd behaviour on network - RSTP and BDPU sanity check

BenBeck — Mon, 28 Oct 2024 13:01:43 GMT

Hey there,

A couple items:

1) For spanning tree best practices, please check this post: https://community.ruckuswireless.com/t5/RUCKUS-Self-Help/Configuring-RSTP-802-1w-and-Spanning-Tree-Best-Practices-on-ICX/m-p/36660#M25

2) Please upgrade the 8200s to 10010e (10010a is a bit rough to be blunt)

Regarding the connectivity example, it is very hard to say. I would recommend opening a support case for that and having one of our engineers assist in troubleshooting if it persists on 10010e. The forum is probably not the best bet for that one.

EDIT: I also see rstp hello parameters have been tweaked. Please ensure these match throughout the network or remove them if unnecessary.

Re: Odd behaviour on network - RSTP and BDPU sanity check

Killerbot — Mon, 28 Oct 2024 14:06:54 GMT

Hi Ben,

Got to site and couldn't find anything untoward happening on the network at all but behaviour persisted. Rebooted the core and all came back up without issue. I'll speak to the team onsite and ask that they schedule the update for out of hours asap.

Tested what happend when creating loops and STP would block one and keep the other as a forwarder. I may advise they put bdpuguard on to properly kill the links instead.