Beamflex causing roaming problems with warehouse devices

[jeff_roback]

We’re having a really tough time getting 25 R610’s working in a warehouse environment with Vocollect (Honeywell) T5 devices that are used by the warehouse personnel. After a lot of investigation, review of logs, and packet captures it appears that the issue is around the T5 devices misreading the signal strength of the AP’s and staying on ap’s with poor coverage. The T5 devices are deigned to be used by operators that are driving forklifts around a warehouse so they are extremely sensitive to changes in the RF environment, always looking for a better ap.
Unfortunately they are using the AP beacons to read signal strength, so even if they are connected to an AP with great coverage, when they see a beacon from that ap with a weak signal, so when they see a beacon from an AP that physically does have good coverage but shows as having a weak signal, they assume the operator has moved from that AP and they imediately scan and jump to another ap. This process seems to repeat over and over again, resulting in a terrible experience for the client devices. These T5 devices do use older 802.11 b/g radios, and the newer devices like ipads, iphones, androids and laptops work great. Unfortunately the T5 devices are mission critical so I need a way to get them working reliably. Before Ruckus was in place, the warehouse was using Ubiquity access points, which worked great for the t5’s, just not for anything else... So we’re kind of stuck now... overall it seems like both the Ruckus and the T5 devices are trying to independently solve the same problem but they’re conflicting with each other resulting in terrible device experience. We’ve tried just about every configuration change possible including adjusting signal strength up and down, changes to power management mode on the devices, forcing b, forcing g, bss minrate, smartroam, changing roam thresholds on the device, etc.... but nothing seems to get the devices to believe the rucks signal is as strong as it actually iis because the beacons show lower strength. Does anyone have any experience with client devices that are using beacons to identify signal strength, and have any strategies to prevent them from getting confused by beamflex? Vocollect/Honeywell has asked if we can turn off beamflex, but it seems like that’s built in to the hardware and not changable. I have opened a case with Ruckus tech support but haven’t been able to get past the “Just reboot It” stage... I don’t think they understand we have a fundamental problem with the interaction of the devices and beamflex. Thanks for any thoughts....

[tom_pisacic]

Forgot to mention, the data rate is set for both band to 12Mbits.

[michael_brado]

Unfortunately, the decision to roam is entirely up to the client radio and drivers.

I wouldn't recommend using Ruckus load balancing that would "encourage" clients to roam if an AP has many more clients than those around it.

[jeff_roback]

In our situation this was caused by some very unusual roaming behavior in the client devices.  The devices we are using Vocollect (Honeywell) T5 are very old devices and thus use an older 2.4 GHz radio.   Based on my experience I think beamflex was definitely causing problems for the devices. 

Because the handsets were designed for use while walking around a warehouse, and they were created in 802.11b days, they have some pretty sophisticated code going on for how they detect and roam between access points. My best reverse engineering seemed to indicate that they assume that a decrease in signal strength means that someone is walking away from an AP and that they should roam from it and not try that one again for a while.   And this seemed to work very will with most of AP vendors that don't have much intelligence going on at the AP.  But in a Ruckus environment where the AP's have a lot of RF intelligence going on and beamflex was constantly changing the signal strength for individual APs, the two devices trying to figure each other out was creating havoc.   
I'm not a RF engineer, so this may be complete gibberish, but here's what it looked like to me:  In watching the packet captures and debug logs, the beacons seemed to be causing the wireless devices great difficultly.   Since the beacons are sent out without any client specific beamflexing, the RSSI levels for those would vary greatly from what the client was already seeing.   So the intelligence in the device would use RSSI levels in the beacons as a 'picture' of what the environment looked like and then make very bad choices about when and where to roam.   (The developers did confirm that the driver code utilizes RSSI from beacons to learn about the environment.).   And that accounted for why the devices would roam even when they had great signal strength... because directed packets were coming in a different signal levels than beacons.   

Of course the Ruckus gear was working great for the other devices in the warehouse, so I didn't want to break the environment for everyone else.

Also, I should mention the folks at Honeywell were incredibly helpful!  They jumped right in with me and put their hardware and software people on the problem which was great.

After spending a ton of time with them looking at packet captures and debug logs, we did end up getting this working with the following settings.

Device side:   

1. Set device for = 802.11b only (this was critical.  Even through the devices supported 802.11g, that didn't work reliably).  Setting OFDM only on the AP side didn't work reliably either. 
2. Adjust roam triggers to make the devices wait for a pretty bad signal before roaming.
3. Set devices to only scan channels 1,6,11

Ruckus side:

1. Hide SSID ENABLED  (This was critical.  It seemed that by hiding the SSID in the beacons, the mobile devices didn't see the AP's at constantly changing RSSI levels, so they got less confused).
2. 802.11r, 802.11k, 802.11w disabled
3. DTIM 2 (Manufacturer specific)
4. Directed MC/BC threshold: 1
5. Mgmt TX Rate 1 mbps
6. Reduce the AP output strength so that no more that 1 AP per channel is visible at  -75 RSSI.  

If anyone else is using the vocollect devices... .this is the vocollect device config that ended up working for us with the above ruckus config;

[HKEY_LOCAL_MACHINE\Software\Vocollect\NETWORKD\RadioSettings]
    ModulationMode=b
    PowerMode=dword:0
[HKEY_LOCAL_MACHINE\Comm\SDCCF10G1\Parms\configs\GlobalConfig]
    bLRS=dword:00000421
    RoamTrigger=dword:00000046
    RoamDelta=dword:0000000F

[michael_brado]

Excellent feedback Jeff, thanks.  Hm, all direction broadcast beacons signal strength affected T5 roaming...  Did hiding the SSID alone help the T5s?

[tom_pisacic]

Thanks Jeff, this is the detailed report. 

Since you mentioned to mange 1 AP per channel to be visible at -75 or lower, what is the AP overlap in that case (in percentage)?