ZF7372 High Client Density issue

[wlangeek]

Hi Everyone,

I'm evaluationg the zf7372 as a potential replacement for some 2942's and 7363's that I use in a very high density environment. We typically budget 75-85 stations per radio on the current AP's we have. From the documentation on the 7372 it looks like it can handle 250 stations per radio and I'd really like to present a case for upgrading to reduce protocol overhead by reducing the number of AP's, but I need to get a proof working in the lab first.

I've setup a testing environment in one of my labs, and have only been able to get ~120 devices to associate before the AP stops accepting associations. The logs on the attached ZD1100 show "User [DEVICE_MAC] fails to join WLAN [WLAN_NAME] from AP[AP_MAC]"

I've done a little more snooping using some of the tools I have and it appears that once I hit this ~120 device limit, the newer devices are able to associate, but are immediately (within ~7ms) sent a deauth frame. This testing environment is an open/WEP64 wlan, so I don't think I'm running into memory or other resource issues on the AP during the negotiation phase, but I could be wrong. I've looked at the AP logs and they show that the AP has plenty of memory remaining (60+MB)

I'm running 9.6.0.0.267 on a zf1100 with the same release on the AP.

The logs on the AP show the following errors once I reach the ~120 device range and start to see the deauth frame response to new clients:

Aug 12 23:35:04 RuckusAP user.info kernel: tac_set_station_key(): tac_set_station_key: new key failed
Aug 12 23:35:04 RuckusAP user.info kernel: net80211_tac_cfg_sta_add(): add station {DEVICE_MAC} session key,failed cipher = 2
Aug 12 23:35:04 RuckusAP user.info kernel: tac_set_station_key(): tac_set_station_key: new key failed
Aug 12 23:35:04 RuckusAP user.info kernel: net80211_tac_cfg_sta_add(): add station {DEVICE_MAC} session key,failed cipher = 2
Aug 12 23:35:04 RuckusAP local2.err syslog: Failed add station in processing MSG_MOBILE_CFG_REQ

I've tried disabling all the features I can to narrow down the issue, but I haven't had any luck (background scanning on/off, dropping multicast packets on/off, client load balancing on/off, client fingerprinting on/off, only one wlan active on the AP, etc). I've also ensured that the device limits are set higher than 120 clients so I don't believe I'm running into an issue there. I also don't see any warnings about reaching 90%+ of the AP's capacity as I generally do when I approach the limits on a 7363 device which also leads me to believe that it's a software issue and not a misconfiguration.

Anyone have some pointers on what might be going on here? It's starting to look like a bug in the AP firmware somewhere in the key management subsystem to me.

~WlanGeek

[keith_redfield]

Do you have TKIP enabled by any chance? Switch to AES explicit if so (don't use auto). TKIP abuses the CPU.

[wlangeek]

I'm using wep-64 as most of my device chiptsets have builtin hardware for wep and don't require the use of wpa-supplicant to handle encryption. The CPU utilization seemed to be around 50% according to the logs while I was experiencing this issue. Any chance you could forward the error log to an engineer? This seems like the kinda thing the right individual would look at and either say "hrm...that's interesting...it should never do that" or "hrm...I know just what's going on here".

[keith_redfield]

If you get a case opened I can get it to our escalation team (I would open a case for you but I can't ID you in any of our systems using your forum email...)

-K

[michael_brado]

WlanGeek, your AP can support more than 100 clients, thou WEP is not a regular
client, but requires wifi chip hardware keycache which is limited to 128 slots, and
with some overhead, 112 slots is a more likely limit for WEP (only) clients. You
should find that you can add additional Open Auth (or WPA) type clients after
you've maxed the WEP clients.

Please also be aware that WEP will not be supported in our (very near) future
ZoneDirector/SCG releases of firmware.