ICX7250 Stack : DOT1x sessions gone until reboot

[stephan_schuste]

Hello Colleagues,

after updating our ICX-7250 devices to 08.0.90k, I'm noticing a strange behavior. Some days after a reboot of our Stack (3 devices), the dot1x sessions on all of the stack members, except the 1st one (the master), are gone. If I use the command 

show dot1x session all

I see only sessions of the first stack device (1/1/x). After rebooting the stack, all is back to normal and I see a much longer list of sessions, including ports of all 3 stack devices, as expected.

The log (show logging) shows me that the sessions where closed because of "port down" but the ports are up and also reconnecting a device (cable) is not able to bring the dot1x session back. It stays broken until the stack reboots. Single devices are not affected so far, maybe just because they are always #1 in there own single device stack and can't be affected.

Did anyone have the same issue and is there a firmware with a fix?

kind regards

stephan

[Orlando_Elias]

Hello Stephan, 

I have not seen this issue, but it is something that for sure we should investigate.

I would look at the health of the stack, if it's a linear stack it's more likely to have sync issues due to hardware problems in the fiber links or optics.
In addition, I would suggest some more testing to isolate the problem, like giving a reload to specific units separately, changing the active controller role to a different unit.

Finally, there was an issue in 8090m switching version for which we recommend moving to 8090mc version.

Please let me know your thoughts.

[mohammad_atmazi]

Do upgrade or downgrade for the switch image and test again 

[stephan_schuste]

Thank you so far. I checked the stack status at the time the sessions were broken, but the stack was displayed as healthy. I will try to update to 8090mc next time the problem comes up and will keep you updated.

[stephan_schuste]

Short update: After the third case, I updated to 8090mc version. Unfortunately i got the error again. So I will post bit more information.

#show dot1x session all
---------------------------------------------------------------------------------------------------------------------
Port    MAC               IP(v4/v6)            User              VLAN  Auth      ACL    Session   Age   PAE  
        Addr              Addr                 Name                    State            Time            State
---------------------------------------------------------------------------------------------------------------------
1/1/7   [removed]    N/A                  [removed] 100   permit    None   4634      Ena   AUTHENTICATED
1/1/7   [removed]    N/A                  N/A               194   guest     None   1009445   Ena   HELD
1/1/13  [removed]    N/A                  N/A               194   guest     None   1009429   Ena   HELD
1/1/26  [removed]    N/A                  N/A               194   guest     None   1009430   Ena   HELD
1/1/28  [removed]    N/A                  N/A               4092  init      None   1         N/A   CONNECTING
1/1/37  [removed]   N/A                  N/A               194   guest     None   1009371   Ena   HELD
1/1/38  [removed]    N/A                  N/A               194   guest     None   1009371   Ena   HELD

Stack health looks good:

#show stack
T=11d16h30m29.5: alone: standalone, 😧 dynamic cfg, S: static
ID   Type          Role    Mac Address    Pri State   Comment                   
1  S ICX7250-48P   active  [removed] 128 local   Ready
2  S ICX7250-48P   standby [removed]   0 remote  Ready
3  S ICX7250-48P   member  [removed]   0 remote  Ready

    active       standby                                                       
     +---+        +---+        +---+                                           
     | 1 |2/1--2/1| 2 |2/3--2/1| 3 |                                           
     +---+        +---+        +---+                                           
Standby u2 - protocols ready, can failover
Current stack management MAC is [removed]


#show interfaces stack-ports 

Port       Link    State   Dupl Speed Trunk Tag Pvid Pri MAC             Name
1/2/1      Up      Forward Full 10G   None  No  N/A  0   [removed]                 
2/2/1      Up      Forward Full 10G   None  No  N/A  0   [removed]                 
2/2/3      Up      Forward Full 10G   None  No  N/A  0   [removed]                 
3/2/1      Up      Forward Full 10G   None  No  N/A  0   [removed]

But logging shows some interesting facs:

Nov 18 18:21:18:C:Stack: Stack unit 2 has been deleted from the stack system Nov 18 18:24:21:I:System: Interface ethernet 2/2/3, state up 
Nov 18 18:24:21:I:System: Interface ethernet 2/2/1, state up 
Nov 18 18:24:15:I:Stack: Stack unit 1 has been elected as ACTIVE unit of the stack system 
Nov 18 18:24:15:I:Stack: Stack unit 3 has been added to the stack system 
Nov 18 18:24:15:I:Stack: Stack unit 2 has been added to the stack system 
Nov 18 18:24:11:I:System: Interface ethernet 1/2/1, state up 
Nov 18 18:24:10:I:System: Interface ethernet 1/2/1, state down 
Nov 18 18:24:05:I:System: Interface ethernet 1/2/1, state up 
Nov 18 18:21:34:I:System: Interface ethernet 1/2/1, state down 
Nov 18 18:21:22:I:Stack: Stack unit 1 has been elected as ACTIVE unit of the stack system 
Nov 18 18:21:22:I:Stack: Stack unit 3 has been added to the stack system 
Nov 18 18:21:22:I:Stack: Stack unit 2 has been added to the stack system 
Nov 18 18:21:19:I:System: Interface ethernet 1/2/1, state up 
Nov 18 18:21:18:I:System: Interface ethernet 1/2/1, state down 
Nov 18 18:21:18:I:System: Interface ethernet 3/1/47, state down 
Nov 18 18:21:18:I:System: Interface ethernet 3/1/45, state down 
Nov 18 18:21:18:I:System: Interface ethernet 3/1/38, state down 
Nov 18 18:21:18:I:System: Interface ethernet 3/1/35, state down 
Nov 18 18:21:18:I:STP: VLAN 4094 Port 3/1/35 STP State -> DISABLED (PortDown) 
Nov 18 18:21:18:I:STP: VLAN 4094 Port 3/1/35 STP State -> BLOCKING (DOT1wTransition) 
Nov 18 18:21:18:I:STP: VLAN 4094 Port 3/1/35 STP State -> BLOCKING (PortDown) 
Nov 18 18:21:18:I:System: Interface ethernet 3/1/27, state down 
... 
Nov 18 18:21:18:I:System: Interface ethernet 3/2/1, state down 
Nov 18 18:21:18:C:Stack: Stack unit 3 has been deleted from the stack system 
Nov 18 18:21:18:I:System: Interface ethernet 2/1/46, state down 
... 
Nov 18 18:21:18:I:System: Interface ethernet 2/2/1, state down 
Nov 18 18:21:18:C:Stack: Stack unit 2 has been deleted from the stack system

so it seems that the stack were broken and rebuilt. All came back except the dot1x sessions / possibility to create those on switch 2+3.

Hope this helps. Will remove the stack tomorrow and use them as single devices from now on.

kind regards

stephan