Radius authentication stopped working on ICX 7250

[stephan_schuste]

Dear Colleagues,

I have a couple of ICX 7250, all configured quite similar. On one of them radius authentication stopped working. I just can't login by using my AD credentials anymore. Only local auth is working. I checked on the Firewall and saw that the switch is not even sending packets to the Radius Server (windows nps).

I started debugging on that switch and to compare it on a working switch as well.
The broken one shows me a debug log like this:

Debug: Jul 24 15:36:10 Function aaa_authenticate_start_internal, session:17, cloud_enable:0, login:username, SSH/Con/Tel(get_ui_session_idx):1, cloudUI(get_ui_session_idx):0, AAA method:2
Debug: Jul 24 15:36:10 RADIUS authentication for context 17
Debug: Jul 24 15:36:10 RADIUS ERROR: Client index 255 out of range.
Debug: Jul 24 15:36:10 Failed to get a RADIUS clientCB; Returning AAA_BUSY for context 17
Debug: Jul 24 15:36:10 Function aaa_authenticate_callback, session:17, cloud_enable:0, login:username, SSH/Con/Tel(get_ui_session_idx):1, cloudUI(get_ui_session_idx):0, AAA method:2, rc:5
Debug: Jul 24 15:36:10 Authentication Timeout or busy, rollover to next server or method.
Debug: Jul 24 15:36:10 Authentication: Next method is Local.
Debug: Jul 24 15:36:10 Function aaa_authenticate_callback, session:17, cloud_enable:0, login:username, SSH/Con/Tel(get_ui_session_idx):1, cloudUI(get_ui_session_idx):0, AAA method:3, rc:4
Debug: Jul 24 15:36:10 Authentication Timeout or busy, rollover to next server or method.
Debug: Jul 24 15:36:10 Authentication: No more method left.
Debug: Jul 24 15:36:10 Function aaa_authenticate_callback, session:17, cloud_enable:0, login:username, SSH/Con/Tel(get_ui_session_idx):1, cloudUI(get_ui_session_idx):0, AAA method:0, rc:3
Debug: Jul 24 15:36:10 Authentication failed.
Debug: Jul 24 15:36:10 aaa_send_aaa_response()..session 17, err_code=3 deferred_response_id=f81a89
Debug: Jul 24 15:36:29 Authentication Stopped, AAA Session context is 17.
Debug: Jul 24 15:36:29 ====AAA: Cleanup session 17 information.

on a switch with working radius auth it looks like this:

Debug: Jul 24 15:39:57 ====AAA: Cleanup session 17 information.
Debug: Jul 24 15:39:57 Function aaa_authenticate_start_internal, session:17, cloud_enable:0, login:username, SSH/Con/Tel(get_ui_session_idx):1, cloudUI(get_ui_session_idx):0, AAA method:2
Debug: Jul 24 15:39:57 RADIUS authentication for context 17
Debug: Jul 24 15:39:57 RADIUS ERROR: Client index 255 out of range.
Debug: Jul 24 15:39:57 Reseting RADIUS Client structure
Debug: Jul 24 15:39:57 RADIUS: Reset client 0, Session type 1, Total number of active clients=1
Debug: Jul 24 15:39:57 AAA: Open RADIUS UDP port
Debug: Jul 24 15:39:57 RADIUS message received from server of len 127.
Debug: Jul 24 15:39:57 Radius secret len ?25, total len ?127
Debug: Jul 24 15:39:57 RADIUS Timer cancelled for client 0.
Debug: Jul 24 15:39:57 RADIUS server ACCEPTed request
Debug: Jul 24 15:39:57 Function aaa_authenticate_callback, session:17, cloud_enable:0, login:username, SSH/Con/Tel(get_ui_session_idx):1, cloudUI(get_ui_session_idx):0, AAA method:2, rc:1
Debug: Jul 24 15:39:57 Authentication successful.
Debug: Jul 24 15:39:57 aaa_send_aaa_response()..session 17, err_code=1 deferred_response_id=a86a8
Debug: Jul 24 15:39:57 Closing RADIUS UDP port
Debug: Jul 24 15:39:57 RADIUS: radius_authenticate_stop for client Idx 0. Actv Clients left 0
Debug: Jul 24 15:39:57 Reseting RADIUS Client structure
Debug: Jul 24 15:39:57 Authorization status - accept.
Debug: Jul 24 15:39:57 aaa_send_aaa_response()..session 17, err_code=1 deferred_response_id=a86aa

 So both are showing the Error "RADIUS ERROR: Client index 255 out of range." but the working one is performing a "Reseting RADIUS Client structure" after that and the broken one don't.

Both switches are running SW: Version 08.0.90kT211

best
stephan

[BenBeck]

Hey @stephan_schuste 

The 8090 code train is no longer being worked on. It is very likely you are hitting an old bug. I would recommend you upgrade to 8095n (current stability release) or 8095p.

[Chandini]

Hi Stephan

Thank you for reaching us

I had come across similar radius logs. Reload fixed the issue in the scenario I had worked on. You can try to reload the switch and see if it fixes the issue for you. 

Take a backup of running configuration of the switch before you try to reload. 

Thanks 

[stephan_schuste]

A reboot did not help. I set a new radius client secret on Radius-Server and Switch side and this worked for a few days. Now its broken again but different than before. If I login by ssh with a radius or local account, the terminal gets stuck. So login is currently not possible by using ssh. I'm gonna update the switch to the recommended firmware version.

[Chandini]

Hi Stephan

Noted and Thanks for reaching out to us on portal.