ZF7372 High Client Density issue

[wlangeek]

Hi Everyone,

I'm evaluationg the zf7372 as a potential replacement for some 2942's and 7363's that I use in a very high density environment. We typically budget 75-85 stations per radio on the current AP's we have. From the documentation on the 7372 it looks like it can handle 250 stations per radio and I'd really like to present a case for upgrading to reduce protocol overhead by reducing the number of AP's, but I need to get a proof working in the lab first.

I've setup a testing environment in one of my labs, and have only been able to get ~120 devices to associate before the AP stops accepting associations. The logs on the attached ZD1100 show "User [DEVICE_MAC] fails to join WLAN [WLAN_NAME] from AP[AP_MAC]"

I've done a little more snooping using some of the tools I have and it appears that once I hit this ~120 device limit, the newer devices are able to associate, but are immediately (within ~7ms) sent a deauth frame. This testing environment is an open/WEP64 wlan, so I don't think I'm running into memory or other resource issues on the AP during the negotiation phase, but I could be wrong. I've looked at the AP logs and they show that the AP has plenty of memory remaining (60+MB)

I'm running 9.6.0.0.267 on a zf1100 with the same release on the AP.

The logs on the AP show the following errors once I reach the ~120 device range and start to see the deauth frame response to new clients:

Aug 12 23:35:04 RuckusAP user.info kernel: tac_set_station_key(): tac_set_station_key: new key failed
Aug 12 23:35:04 RuckusAP user.info kernel: net80211_tac_cfg_sta_add(): add station {DEVICE_MAC} session key,failed cipher = 2
Aug 12 23:35:04 RuckusAP user.info kernel: tac_set_station_key(): tac_set_station_key: new key failed
Aug 12 23:35:04 RuckusAP user.info kernel: net80211_tac_cfg_sta_add(): add station {DEVICE_MAC} session key,failed cipher = 2
Aug 12 23:35:04 RuckusAP local2.err syslog: Failed add station in processing MSG_MOBILE_CFG_REQ

I've tried disabling all the features I can to narrow down the issue, but I haven't had any luck (background scanning on/off, dropping multicast packets on/off, client load balancing on/off, client fingerprinting on/off, only one wlan active on the AP, etc). I've also ensured that the device limits are set higher than 120 clients so I don't believe I'm running into an issue there. I also don't see any warnings about reaching 90%+ of the AP's capacity as I generally do when I approach the limits on a 7363 device which also leads me to believe that it's a software issue and not a misconfiguration.

Anyone have some pointers on what might be going on here? It's starting to look like a bug in the AP firmware somewhere in the key management subsystem to me.

~WlanGeek

[jon_prouty]

WlanGeek, I am concerned of your initial step of trying to reduce ap count to open up protocol overhead? Are you talking about the management and control frames? If you are having traffic issues with your ap's separate the ap's into different vlans to reduce the amount of traffic in each vlan. Keep your clients separate from your management vlans and the management traffic should be a negligible issue if one at all. Your clients will have a much easier time if you can spread the clients out between more radios, not fewer radios at higher client density! Remember this is a shared medium and only one client can talk to one radio at a time, throughput of a radio with 250 clients attached to it has got to be a trickle and that would happen even with ac and 80 MHz channels. That is setting yourself up for failure in my opinion. Do you really want 250 clients on each radio? Not this guy!

[eizens_putnins]

Hi, Jon Prouty,
Actually exactly because it is shared medium you want to have less APs supporting more clients -- because if you have multiple APs, on the same channel and near to each other, only one radio anyway can transmit at the time, and there are a lot of additional overheads, primarily related with detection if channel is free to transmit, contention and management traffic of every AP. This overheads may go up to 50% of available bandwidth.
And if you try to say that it is better to use 2 or more APs on different channels, it is normally done already -- we have already situation when there are no unused channels available to use for additional capacity, so you must manage as much efficiently as possible what you have available, and it is much easier with less high-density APs, than with many low density APs.
And with 802.11ac we soon will have situation on 5Ghz quit similar to what we have in 2,4 GHz now -- and efficient management of available spectrum is extremely important to get satisfactory results.

[jon_prouty]

Hi Eizens, the way you get by co-channel interference between ap's is properly surveying before you place your access point. If you are running into areas that can no longer place any more access points without causing co-channel interference then yes you have to start replacing older model ap's with newer ap's that can handle the higher densities of clients. The overall fact here that I was trying to point out is the clients will experience better overall performance with fewer clients per radio with all clients (abgnac).

[eizens_putnins]

Hi, Jon,
You probably leave in some happy land, where you can install AP and it will be the only on the channel.
During last 3 years I haven't seen any installation in a city, where I can't see on the every channel at least 3-4 APs, and what is even worth -- most of them are out of your control at all. Only exception are metal buildings, which are shielding everything coming from outside, but than everything which is inside creates even more interference.
Especially much harm cause cheap home 802.11N gear, configured to use 40 MHz channels on 2,4 GHz, and old 802.11b/g APs. You don't need to have 30 dbm level to prevent transmission, it is enough that client can hear -70dbm level of neighbor AP transmission.
The only way to fight this till now is using Ruckus APs with they automatically adjusted directional antenna arrays -- if they doesn't work, my practice shows that no other WiFi gear work.
Recently we had a case -- Wi-Fi (brand new Apple Exterme Air-port and SMB APs from Cisco-Linksys and old 802.11g HP Procurve) provided 10-20 KB/s download speeds in 10 meters of AP in the same room, without any obstacles. Pwner of brand new Apple Air was quit unhappy.
Survey has demonstrated 20x APs with RSSI more that -50 dbm. One was public Hot-spot AP on the next building, actually in front of window, providing -30 dbm RSSI, and there have been at least one AP with RSSI higher that -40 dbm on every channel. There was no non-wifi interference, as we checked using spectrum analyzer, jut spectrum was used up to 80-100% average.
As a last hope I provided Ruckus ZF7982 AP, and never get it back -- as Apple Air have got on spot decent 50 MB/s download speed.

And if you really want more performance, you must get read-off 802.11b clients and disable non-OFDM modulations, this would improve everything except for antique devices.

[eizens_putnins]

It would be really peace of cake, if our Wi-Fi installations would be just islands in a non-wi-fi-ed world, as it was 10 years ago. SOHO equipment unnumbered long ago any enterprise equipment, so whenever we look, we can feel them...
So 2,4 GHz band is going to be abandoned because it is overloaded...
But I can say for sure - it is not the end, no -- it is actually the beginning... Now we have 802.11ac standard in place, so soon most devices will support at least some limited 802.11ac version, and 5GHz band will be as much full of garbage, as 2,4GHz is now.
I expect that actually in most dense enterprise applications 802.11ac APs will use 40 MHz channels, to avoid interference from neighbor APs and outside devices, and real improvement of speed will come not from plain numbers (such as 160MHZ channel or 256 QAM modulation), but from better chipsets, better firmware, better active antennas, better radio-management and so on. And it will not come as easy as we would like.
And surveys have now to be done not just for coverage and inter-AP interference, but for capacity, taking into account Wi-Fi system intelligence and environment.
It is much more complicated, so hopefully new tools are coming to help us to do it soon.
Hope it helps - we have to discuss this issues quit often now, when 802.11ac gear comes more and more around.