topic R720 DHCP client bug in Unleashed

R720 DHCP client bug

david_fuchs — Wed, 30 Dec 2020 19:18:47 GMT

I have a R720 running Unleashed, which gets its IP address from a Mikrotik router via DHCP, with a lease time of 1hr. The DHCP client implementation has a weird issue, where the DHCP lease is released before it is up, but the address continues being used. This is what I see in the router's logs:

Dec/30/2020 10:18:41 LAN assigned 10.1.10.250 to 24:79:2A:XX:XX:XX

Dec/30/2020 10:03:23 LAN deassigned 10.1.10.250 from 24:79:2A:XX:XX:XX

Dec/30/2020 09:18:36 LAN assigned 10.1.10.250 to 24:79:2A:XX:XX:XX

Dec/30/2020 09:03:08 LAN deassigned 10.1.10.250 from 24:79:2A:XX:XX:XX

etc, etc - with this exact pattern repeating every hour.

I suspect the 720 is attempting to refresh the lease after 45 minutes, but instead ends up simply terminating it. I have an R710 and R510 also, which do not exhibit this behavior. (In fact no other device on my network is behaving like this, which leads me to believe this is a Unleashed issue, rather than a Mikrotik one.)

For the most part the bug is benign (the R720 continues to use the DHCP assigned IP after terminating the lease), but I do occasionally experience spurious system reboots which I believe are the result of address collisions resulting from the use of an IP address after the lease has been terminated.

Making the lease for all the unleashed APs static seems to be a usable workaround, but a buggy DHCP client in a networking product that can lead to address collisions is something Ruckus should fix...

Unleashed version 200.9.10.4.212.

EDIT: Unlike I originally suspected this will not lead to address collisions. The reboots are due to something else.

Re: R720 DHCP client bug

dave_raines_86z — Wed, 30 Dec 2020 20:17:59 GMT

Are these Apple devices on OS 14.2 or higher with private IP enables on the SSID whereas it spoofs a fake mac address?

Re: R720 DHCP client bug

david_fuchs — Thu, 31 Dec 2020 01:18:17 GMT

No, these aren't some random devices connected to the AP - it's the AP itself that is misbehaving.

Re: R720 DHCP client bug

eizens_putnins — Sat, 02 Jan 2021 21:32:58 GMT

This sounds impossible -- normally router always assigns same IP to same MAC, if this device is up, never assigns to same device different address, and also it never assigns that previously used IP address to any different device, as far as AP was not off-line for a long period of time. I have many installations, where Ruckus AP get IP from Mikrotik routers, but not exactly same configuration -- R720 in unleashed configuration, we mostly use only vSZ managed different models.

Anyway, as a first step I would propose to upgrade Mikrotik router to newest firmware, as what you describe doesn't look an AP issue -- router should never assign IP to any device, if such IP is reachable on network. It should be marked as bad IP and not used instead. Upgrade is done by couple of clicks - in menu System/packages click on "download and upgrade".

I have seen issues with DHCP on some networks, when ARP proxy feature was enabled on WLAN, as DHCP server understood that as assigned address, so check if this option is checked and try disable it.

Fact that different model APs doesn't show same issue may be related with fact that R720 consumes much more power and uses 2.5Gbps-capable port.

Next-- how is your AP powered? Reboots and any other issues are very possible if there is not enough power for AP -- if you use 802.3at ports, you need to disable second Ethernet port and USB port. I would recommend to use at least 50W PoE injector.

Another thing to check is Ethernet port speed and connection quality. R720 supports 2.5GBps port speed, and with low quality cable connection, or low quality PoE injector, you may easy get issues. For example, using non-gigabit PoE injector guarantees AP reboots with Kernel panic messages in the log...

Hope it helps,

Re: R720 DHCP client bug

david_fuchs — Mon, 04 Jan 2021 20:07:49 GMT

Thanks!

The occasional reboots I'm experiencing are very annoying, but may have nothing to do with the DHCP issue I reported.

Regardless of the reboot issue, the fact that the R720 releases its IP address but then keeps using it anyway is definitely not a correct implementation of DHCP, and Ruckus needs to fix it. I am reasonably sure that this is a Ruckus issue, and not a Mikrotik one, as no other device on my network - including 2 other Ruckus APs - behave this way. I agree that it's unlikely that the DHCP server will re-assign the IP to a different device in the short window where the AP is using it illegitimately (unless something really funky is happening, like some other device impersonating that same MAC - unlikely), but still - it's broken.

My access points are connected to a POE switch, via 2.5G 802.3at (for the 720). Load is reported as somewhere in the 8-10W range, well within the PoE+ specs and well below the 300W budget of the switch. The mikrotik router is up-to-date. The AP's reboot reason is "last disconnected reason [Heartbeat Loss]" - again, any connection to the DHCP issue was just wild speculation on my part.

Re: R720 DHCP client bug

eizens_putnins — Thu, 07 Jan 2021 12:50:55 GMT

If your switch reports power used by R720 as 8W, something is seriously wrong with your setup, and R720 runs in some fail-save 802.3af-compatible mode. This most probably is the very reason of your unexpected reboots -- Ruckus APs almost never reboot without reason.

Max power consumption is always more than average, of cause, but not that much, and anyway insufficient power at boot switches AP to lower performance mode in the best case, in the worst -- prevents booting or results in different errors.

Max power consumption for R720 is:

in 802.3af mode 12.95 W

802.3at mode 25.5 W

Full mode 33.5 W

So I strongly recommend to provide proper power to R720 APs, or at least configure APs in compatible with 802.3at power way from controller (disable second Ethernet port and USB port). Only when this is done, it makes sense to look for any other issues (which most probably will disappear than).

Using R720 with insufficient power just doesn't make sense -- PoE injectors doesn't cost much, and I would definitely not want to limit capability of rather expensive APs because of low power. If you can't provide proper power, it would be better to use R510 APs (which are much cheaper), as this would provide probably same or even better performance, than you have now with R720 in degradeted mode.

Re: R720 DHCP client bug

david_fuchs — Thu, 07 Jan 2021 20:11:29 GMT

Thanks for the tips! The 720 claims it gets power via 802.3at:

Re: R720 DHCP client bug

david_fuchs — Thu, 07 Jan 2021 20:40:46 GMT

Also, the 8-10W range seems to be about right.

Re: R720 DHCP client bug

david_fuchs — Thu, 07 Jan 2021 22:26:25 GMT

It gets weirder... I changed the master AP to the R710 - lo and behold, now that device is exhibiting the same weird DHCP behavior.

I noticed that both APs use the same hostname (the name configured under "Admin & Services -> System -> System Info", rather than the individual name set for each AP under "Access Points -> Edit -> General -> Device Name"). Maybe that is confusing the DHCP server...

Re: R720 DHCP client bug

david_fuchs — Fri, 08 Jan 2021 07:37:16 GMT

Yeah, Ruckus' DHCP client implementation is buggy. Or at least, strange.

For whatever reason, the master AP - and only the master - sends a bunch of DHCPDISCOVER messages when it already has an active lease. The DHCP server dutifully responds with offers, which the AP ignores. Mikrotik's DHCP server interprets this as the client no longer being bound (it still respects the original lease time though, so my initial assumption that this could lead to address collisions was wrong).

Why does it broadcast DHCPDISCOVERs when it is already bound?

Re: R720 DHCP client bug

david_fuchs — Fri, 08 Jan 2021 22:15:38 GMT

After wasting waaay too much time with this, my conclusion is that Unleashed does not like long-ish DHCP lease times.

Playing around with the lease times, I found that short (10min) lease times cause no problems. With a 2hr lease time, the master AP actually let the lease expire and kept using the expired IP for ~30 minutes before finally sending another DHCP request.

Pretty shoddy for a $1200 "enterprise grade" networking product.

Re: R720 DHCP client bug

david_fuchs — Mon, 11 Jan 2021 19:21:32 GMT

Sigh. The reboots, after all, do appear to be caused by Ruckus' faulty DHCP implementation, as a result of "heartbeats lost" events.

I see "heartbeats lost" events pretty much exactly the same time every hour, and, sure enough, they correspond exactly to when the master APs lease expire.

Similar issues have already been reported here and here in the past.

I have $10 gadgets that can do DHCP correctly. As already stated - a faulty DHCP implementation is shoddy for an "enterprise grade" networking appliance, and the fact that there are 2 year old reports of this same issue and Ruckus can't be bothered to fix the bug is beyond disappointing.