OSPF LSA Issue ICX7150

[Vladimir_Kos]

Model - ICX7150-C12-2X10GR

Version - 08.0.95rT213

Each time a single port withing a vlan changes operational state, the VE initiates an LSA removal form the tree at the same time that it adds the LSA to the tree. This happens simultaneously and if there is an interface flapping due to a damaged cable, it creates excessive LSA churn that create random pauses in traffic.

[BenBeck]

Hey Shane, 

Thanks for the context. I would get a case opened once support is straightened out. The mention of CLI delay makes me think we may have a CPU spike (which could cause L3 instability). Have we checked that by chance? I did run through known defects since 8095r (and in 8095x). I do not see anything in the realm of OSPF at all. I would *think* this would be a very hot topic.

[shane_owens_i88]

We haven't seen the CPU spike, at least not in the history. What we think is happening is the rapid OSPF flap is causing a temporary pause of routing for OSPF convergence.  We have a fair amount of data captured, but have restrictions on sharing it as this is a government agency. I used to run a test lab and was part of Foundry and Junipers 10Gig Ethernet test clients a long time ago. We will get enough data to at least point engineering in the right direction to find the root cause. Just running to forum route for now to see if anyone else has seen this behavior. I do this it takes a very specific case, or frequency of interface flapping, to create it.

[shane_owens_i88]

Ben, I think Vladimir figured out what is causing it but the why seems like a bug.

In testing we have a 7250 running 8.x code having 4 100Mbps device connections.
We also have a 7150 running 9.x code which has 2 100Mbps connections and 1 10Mbps connection. All tested connections were assigned to VLAN 4 with ve4 configured for routing ospf passive
When we look at the ve for the local LAN (ospf passive) on the 7250 we see a cost of 10 applied in OSPF and this doesn't seem to change when we take interfaces up or down. However, testing a different 7250 running 8.x code with 5 interfaces the initial cost was 11 and when we disabled one the cost dropped to 10.

On the 7150 the cost with all 3 interfaces enabled is 22, if we disable 1 the cost drops to 20 which is the inverse if it were calculated using total bandwidth against a reference. This drop happens whether we drop to 10Mbps or the 100Mbps interface so it's not an aggregate of individual costs, or a calculation vs. configured reference bandwidth
This is where we can't seem to figure out the logic to the calculation, but it does cause the LSA to generate because of the cost change. This flood causes all devices to recalculate which I think is where our occasional pause comes in.

To replicate this all you need to do is have a switch running 9.x code and create at least 3 interfaces (5 interfaces if you are running 8.x code apparently). configure OSPF and make the VE passive. Then show ip ospf interface ve 4 and look at the cost. Disable one of the interfaces and look at the cost again. If it changed then you have replicated this issue

[shane_owens_i88]

Here is an example of what we see and this seems to churn on every flap. 

To see it the easiest method is to do a "show ip route | i s" and that will filter all routes that have updated in less than and hour.

Device that holds the VE:
SSH@MH4#sh log
Syslog logging: enabled ( 0 messages dropped, 0 flushes, 701 overruns)
Buffer logging: level ACDMEINW, 1000 messages logged
level code: A=alert C=critical D=debugging M=emergency E=error
I=informational N=notification W=warning
Static Log Buffer:
Mar 06 01:17:08:I:System: Stack unit 1 Power supply 1 is up
Apr 24 17:56:45:I:System: Stack unit 1 Power supply 2 is up
Dynamic Log Buffer (1000 lines):
May 14 07:48:19:I:System: Interface ethernet 1/1/5, line protocol down
May 14 07:48:19:I:System: Interface ethernet 1/1/5, state down
May 14 07:48:19:I:STP: VLAN 4 Port 1/1/5 STP State -> DISABLED (PortDown)
May 14 07:48:19:I:STP: VLAN 4 Port 1/1/5 STP State -> LISTENING (PortDown)
May 14 07:48:18:I:STP: VLAN 4 Port 1/1/5 STP State -> LISTENING (MakeFwding)
May 14 07:48:18:I:System: Interface ethernet 1/1/5, state up
May 14 07:47:52:I:System: Interface ethernet 1/1/5, line protocol down
May 14 07:47:52:I:System: Interface ethernet 1/1/5, state down
May 14 07:47:52:I:STP: VLAN 4 Port 1/1/5 STP State -> DISABLED (PortDown)
May 14 07:47:52:I:STP: VLAN 4 Port 1/1/5 STP State -> LISTENING (PortDown)
May 14 07:47:52:I:STP: VLAN 4 Port 1/1/5 STP State -> LISTENING (MakeFwding)
SSH@MH4#sh ip route 10.32.48.0
Type Codes - B:BGP D:Connected O:OSPF R:RIP S:Static; Cost - Dist/Metric
BGP Codes - i:iBGP e:eBGP
OSPF Codes - i:Inter Area 1:External Type 1 2:External Type 2
STATIC Codes - v:Inter-VRF
Destination Gateway Port Cost Type Uptime
1 10.32.48.0/28 DIRECT ve 4 0/0 D 69d10h
SSH@MH4#sh clock
07:49:13.453 Eastern Thu May 14 2026
SSH@MH4#sh cpu
68 percent busy, from 5999684 sec ago
It has been more than 50 hours since last call, the CPU utilization
data collected may have wrapped around and resulted in high
utilization
1 sec avg: 1 percent busy
5 sec avg: 1 percent busy
60 sec avg: 2 percent busy
300 sec avg: 2 percent busy
SSH@MH4#sh int brie
Port Link State Dupl Speed Trunk Tag Pvid Pri MAC Name
1/1/1 Up Forward Full 100M None No 4 0 8c7a.153b.e6f4
1/1/2 Up Forward Full 100M None No 4 0 8c7a.153b.e6f4
1/1/3 Up Forward Full 100M None No 4 0 8c7a.153b.e6f4
1/1/4 Up Forward Full 100M None No 4 0 8c7a.153b.e6f4
1/1/5 Down None None None None No 4 0 8c7a.153b.e6f4
<snip>

SSH@MH4#sh ver
Copyright (c) Ruckus Networks, Inc. All rights reserved.
UNIT 1: compiled on Nov 8 2025 at 09:21:20 labeled as GZR10010g_cd5
(64804492 bytes) from Primary GZR10010g_cd5.bin (UFI)
SW: Version 10.0.10g_cd5T243
Compressed Primary Boot Code size = 1081856, Version:10.2.12T245 (gzu10212)
Compiled on Wed Jul 9 06:02:56 2025

Upstream router:
SSH@MH2#sh ip route 10.32.48.0
Type Codes - B:BGP D:Connected O:OSPF R:RIP S:Static; Cost - Dist/Metric
BGP Codes - i:iBGP e:eBGP
OSPF Codes - i:Inter Area 1:External Type 1 2:External Type 2
STATIC Codes - v:Inter-VRF
Destination Gateway Port Cost Type Uptime
1 10.32.48.0/28 10.45.132.213 e 1/3/1 110/61 O 0m9s
SSH@MH2#sh clock
07:49:08.926 Eastern Thu May 14 2026
SSH@MH2#sh cpu
75 percent busy, from 24182150 sec ago
It has been more than 50 hours since last call, the CPU utilization
data collected may have wrapped around and resulted in high
utilization
1 sec avg: 1 percent busy
5 sec avg: 2 percent busy
60 sec avg: 1 percent busy
300 sec avg: 2 percent busy
SSH@MH2#sh ver
Copyright (c) Ruckus Networks, Inc. All rights reserved.
UNIT 1: compiled on Jun 27 2025 at 08:27:22 labeled as GZR09010k
(65955864 bytes) from Secondary GZR09010k.bin (UFI)
SW: Version 09.0.10kT243
Compressed Secondary Boot Code size = 1081856, Version:10.1.27T245 (gzu10127)
Compiled on Thu Feb 6 23:32:28 2025