Seeing almost constant "...joins with uptime [xxxxxx] s and last disconnected reason [Heartbeat Loss]. on 2x r710s

[kevin_vasko]

I'm seeing a lot of error messages in my logs
Jun  8 23:08:47 Downstairs syslog: eventd_to_syslog():AP[Downstairs@2c:c5:d3:21:46:60] joins with uptime [110595] s and last disconnected reason [Heartbeat Loss] 

I have two Ruckus r710's one upstairs and one downstairs. The downstairs unit is the master in unleashed. I'm constantly seeing this but it doesn't tell me why its losing its heartbeat. I think I'm having issues because all of a sudden my clients seem to glitch out and lose connection if they are connected to one of the units.

Any suggestions?

Jun  8 23:08:47 Downstairs syslog: eventd_to_syslog():AP[Downstairs@2c:c5:d3:21:46:60] joins with uptime [110595] s and last disconnected reason [Heartbeat Loss] 
Jun  8 23:08:48 Downstairs syslog: eventd_to_syslog():AP[Upstairs@2c:c5:d3:1a:7f:20] joins with uptime [90809] s and last disconnected reason [Heartbeat Loss] 
Jun  9 01:08:57 Downstairs syslog: eventd_to_syslog():AP[Downstairs@2c:c5:d3:21:46:60] joins with uptime [117805] s and last disconnected reason [Heartbeat Loss] 
Jun  9 01:09:00 Downstairs syslog: eventd_to_syslog():AP[Upstairs@2c:c5:d3:1a:7f:20] joins with uptime [98020] s and last disconnected reason [Heartbeat Loss] 
Jun  9 03:08:37 Downstairs syslog: eventd_to_syslog():AP[Downstairs@2c:c5:d3:21:46:60] joins with uptime [124985] s and last disconnected reason [Heartbeat Loss] 
Jun  9 03:08:41 Downstairs syslog: eventd_to_syslog():AP[Upstairs@2c:c5:d3:1a:7f:20] joins with uptime [105201] s and last disconnected reason [Heartbeat Loss] 
Jun  9 05:08:47 Downstairs syslog: eventd_to_syslog():AP[Downstairs@2c:c5:d3:21:46:60] joins with uptime [132195] s and last disconnected reason [Heartbeat Loss] 
Jun  9 05:08:52 Downstairs syslog: eventd_to_syslog():AP[Upstairs@2c:c5:d3:1a:7f:20] joins with uptime [112412] s and last disconnected reason [Heartbeat Loss] 
Jun  9 07:08:28 Downstairs syslog: eventd_to_syslog():AP[Downstairs@2c:c5:d3:21:46:60] joins with uptime [139376] s and last disconnected reason [Heartbeat Loss] 
Jun  9 07:08:33 Downstairs syslog: eventd_to_syslog():AP[Upstairs@2c:c5:d3:1a:7f:20] joins with uptime [119592] s and last disconnected reason [Heartbeat Loss] 
Jun  9 09:08:39 Downstairs syslog: eventd_to_syslog():AP[Downstairs@2c:c5:d3:21:46:60] joins with uptime [146587] s and last disconnected reason [Heartbeat Loss] 
Jun  9 09:08:44 Downstairs syslog: eventd_to_syslog():AP[Upstairs@2c:c5:d3:1a:7f:20] joins with uptime [126803] s and last disconnected reason [Heartbeat Loss] 
Jun  9 11:08:24 Downstairs syslog: eventd_to_syslog():AP[Upstairs@2c:c5:d3:1a:7f:20] joins with uptime [133983] s and last disconnected reason [Heartbeat Loss] 
Jun  9 11:08:50 Downstairs syslog: eventd_to_syslog():AP[Downstairs@2c:c5:d3:21:46:60] joins with uptime [153798] s and last disconnected reason [Heartbeat Loss] 
Jun  9 13:08:31 Downstairs syslog: eventd_to_syslog():AP[Downstairs@2c:c5:d3:21:46:60] joins with uptime [160979] s and last disconnected reason [Heartbeat Loss] 
Jun  9 13:08:34 Downstairs syslog: eventd_to_syslog():AP[Upstairs@2c:c5:d3:1a:7f:20] joins with uptime [141193] s and last disconnected reason [Heartbeat Loss] 
Jun  9 15:08:42 Downstairs syslog: eventd_to_syslog():AP[Downstairs@2c:c5:d3:21:46:60] joins with uptime [168190] s and last disconnected reason [Heartbeat Loss] 
Jun  9 15:08:44 Downstairs syslog: eventd_to_syslog():AP[Upstairs@2c:c5:d3:1a:7f:20] joins with uptime [148403] s and last disconnected reason [Heartbeat Loss] 
Jun  9 17:08:23 Downstairs syslog: eventd_to_syslog():AP[Downstairs@2c:c5:d3:21:46:60] joins with uptime [175370] s and last disconnected reason [Heartbeat Loss] 
Jun  9 17:08:24 Downstairs syslog: eventd_to_syslog():AP[Upstairs@2c:c5:d3:1a:7f:20] joins with uptime [155583] s and last disconnected reason [Heartbeat Loss] 
Jun  9 19:08:34 Downstairs syslog: eventd_to_syslog():AP[Downstairs@2c:c5:d3:21:46:60] joins with uptime [182581] s and last disconnected reason [Heartbeat Loss] 
Jun  9 19:08:39 Downstairs syslog: eventd_to_syslog():AP[Upstairs@2c:c5:d3:1a:7f:20] joins with uptime [162798] s and last disconnected reason [Heartbeat Loss] 
Jun  9 21:08:20 Downstairs syslog: eventd_to_syslog():AP[Upstairs@2c:c5:d3:1a:7f:20] joins with uptime [169979] s and last disconnected reason [Heartbeat Loss] 
Jun  9 21:08:44 Downstairs syslog: eventd_to_syslog():AP[Downstairs@2c:c5:d3:21:46:60] joins with uptime [189791] s and last disconnected reason [Heartbeat Loss] 
Jun  9 23:08:24 Downstairs syslog: eventd_to_syslog():AP[Downstairs@2c:c5:d3:21:46:60] joins with uptime [196971] s and last disconnected reason [Heartbeat Loss] 
Jun  9 23:08:30 Downstairs syslog: eventd_to_syslog():AP[Upstairs@2c:c5:d3:1a:7f:20] joins with uptime [177190] s and last disconnected reason [Heartbeat Loss] 
Jun 10 01:08:34 Downstairs syslog: eventd_to_syslog():AP[Downstairs@2c:c5:d3:21:46:60] joins with uptime [204181] s and last disconnected reason [Heartbeat Loss] 
Jun 10 01:08:42 Downstairs syslog: eventd_to_syslog():AP[Upstairs@2c:c5:d3:1a:7f:20] joins with uptime [184401] s and last disconnected reason [Heartbeat Loss] 
Jun 10 03:08:14 Downstairs syslog: eventd_to_syslog():AP[Downstairs@2c:c5:d3:21:46:60] joins with uptime [211361] s and last disconnected reason [Heartbeat Loss] 
Jun 10 03:08:23 Downstairs syslog: eventd_to_syslog():AP[Upstairs@2c:c5:d3:1a:7f:20] joins with uptime [191581] s and last disconnected reason [Heartbeat Loss] 
Jun 10 05:08:24 Downstairs syslog: eventd_to_syslog():AP[Downstairs@2c:c5:d3:21:46:60] joins with uptime [218571] s and last disconnected reason [Heartbeat Loss] 
Jun 10 05:08:34 Downstairs syslog: eventd_to_syslog():AP[Upstairs@2c:c5:d3:1a:7f:20] joins with uptime [198792] s and last disconnected reason [Heartbeat Loss] 
Jun 10 07:08:14 Downstairs syslog: eventd_to_syslog():AP[Upstairs@2c:c5:d3:1a:7f:20] joins with uptime [205972] s and last disconnected reason [Heartbeat Loss] 
Jun 10 07:08:34 Downstairs syslog: eventd_to_syslog():AP[Downstairs@2c:c5:d3:21:46:60] joins with uptime [225781] s and last disconnected reason [Heartbeat Loss] 
Jun 10 09:08:15 Downstairs syslog: eventd_to_syslog():AP[Downstairs@2c:c5:d3:21:46:60] joins with uptime [232962] s and last disconnected reason [Heartbeat Loss] 
Jun 10 09:08:24 Downstairs syslog: eventd_to_syslog():AP[Upstairs@2c:c5:d3:1a:7f:20] joins with uptime [213182] s and last disconnected reason [Heartbeat Loss] 
Jun 10 11:08:04 Downstairs syslog: eventd_to_syslog():AP[Upstairs@2c:c5:d3:1a:7f:20] joins with uptime [220362] s and last disconnected reason [Heartbeat Loss] 
Jun 10 11:08:26 Downstairs syslog: eventd_to_syslog():AP[Downstairs@2c:c5:d3:21:46:60] joins with uptime [240173] s and last disconnected reason [Heartbeat Loss] 
Jun 10 13:08:07 Downstairs syslog: eventd_to_syslog():AP[Downstairs@2c:c5:d3:21:46:60] joins with uptime [247354] s and last disconnected reason [Heartbeat Loss] 
Jun 10 13:08:14 Downstairs syslog: eventd_to_syslog():AP[Upstairs@2c:c5:d3:1a:7f:20] joins with uptime [227572] s and last disconnected reason [Heartbeat Loss] 
Jun 10 15:08:18 Downstairs syslog: eventd_to_syslog():AP[Downstairs@2c:c5:d3:21:46:60] joins with uptime [254565] s and last disconnected reason [Heartbeat Loss] 
Jun 10 15:08:24 Downstairs syslog: eventd_to_syslog():AP[Upstairs@2c:c5:d3:1a:7f:20] joins with uptime [234782] s and last disconnected reason [Heartbeat Loss] 
Jun 10 17:07:59 Downstairs syslog: eventd_to_syslog():AP[Downstairs@2c:c5:d3:21:46:60] joins with uptime [261746] s and last disconnected reason [Heartbeat Loss] 
Jun 10 17:08:05 Downstairs syslog: eventd_to_syslog():AP[Upstairs@2c:c5:d3:1a:7f:20] joins with uptime [241963] s and last disconnected reason [Heartbeat Loss] 

[michael_brado]

The long uptimes tell you that the APs are not rebooting.  It appears yoiu have a network issue between the APs
and your controller.  Check intermediate switches/cabling.

Here are a couple related KBAs.
  https://support.ruckuswireless.com/answers/000003945
  https://support.ruckuswireless.com/answers/000001658

[kevin_vasko]

I can only access the top support article (I don't have a subscription as I'm running unleashed).

So I did some more digging and I think what is happening is this.

After doing some more digging I'm thinking it's a problem with the DHCP server or unit doing something funny with its leases. I suspect the ruckus master unit client. I'm using pfsense as my router/dhcp server. It seems to be giving out 2 hour and 24 hour leases. I can't tell you why some clients get 24 and others get 2. It is whatever is setup by default.

The problem seems to stem by the master Ruckus unit not successfully refreshing its DHCP lease of the SAME ip address. I see the unit moving from 192.168.0.62 from yesterday to 192.168.0.74 today (every two hours it changes, so it has moved from 62 to 63, 64,... etc.. This seems to only happen on the master unit. The other unit has a DHCP lease that it keeps renewing and it keeps getting the same IP without a problem.

I feel the issue stems from this scenario.

* Master (Downstairs unit) and slave (upstairs unit) lease their IP and all is well.
* On the downstairs unit, two hours passes, its DHCP lease expires, and then instead of keeping the same IP (e.x 192.168.0.30), a new IP address is requested from the DHCP server
* the Downstairs unit drops from the network and switches IPs
* the slave unit (upstairs unit) loses its heartbeat and rejoins the network as the IP address of the master downstairs has changed

Why this is happening and the master unit is getting new IP addresses beats me. My pfsense is DHCP server leases are full of expired leases from this unit. IThe other Ruckus has not done this once. No other client has done this.

I'm going to test this by setting a static IP address of the downstairs (master) unit. If it stops, I would be willing to bet that is the problem, and why the network drops (and the cause of the heartbeat issues) and clients can't get out to the internet (depending on which AP they are on).

If that is the case, I would suspect the Ruckus master has problems with renewing its DHCP lease (possibly a dhcp client issue on the Ruckus AP firmware?)
I don't have a support license or anything but I'll post back my results when I get home and hopefully the information could be put to good use.

[vitaliy_maruk]

Did you resolve the problem of heartbeat loss? I have the same problem on r610 unleashed firmware 200.9

[syamantakomer]

Hi Vitaliy,

Heartbeat failure is not an issue but a behavior which indicates communication issue between AP to the controller. In case of Unleashed, it is between member APs to master AP.

If you are facing this issue, please check and make sure there is no network connection issue between member APs to master AP. If you are unable to find an issue in the network, please open a case with support and they will help you.

Regards,
Syamantak Omer