multiple heartbeat lost

[fabio_angeletti]

hi, i've installed 4 ap controlled from 1 zone director. the zonedirector is installed in other network. if i deploy only one ap and set director ip manually with the public ip of the zd all go ok. When i install the other 3 ap and set manually zd all ap frequently and randomly disconnect from zd with error "lost heartbeat". All the ap is zoneflex 7343 and connected to dedicate switch and router.

what's the problem?

[monnat_systems]

Fabio angeletti,

lost heartbeat means that AP-ZD communication issue. Quick:

If one AP is ok, how other 3 are NOT, whats different between working and NOT working AP's network or route topology?

I see you menitoned that AP connect to ZD on Public IP, Are these AP connect to ZD from internet, please make sure latency is less than 100 MS,

[fabio_angeletti]

latency < 100ms (50ms)
zd natted 1->1 on public ip and all port open for test
ap in other subnet directly under router with nat and dhcp (the router have 5 port switch) .
if i switch on only 1 ap all is ok . if i switch all or other 1 ap all the up continouse heartbeat lost.
for test i've put all ap under other network with other router and work ok.
the problem is on the router but is confiugrated only with nat and dhcp. no acl!!!!
i look that when connect the ap to the "problematic" router the external ip port on zd ap monitor page is for the first 12223 and other 1010,1011,1012 etc ...when i put all ap in the working sublet/router the external ip port is 10010,10011 etc...

[fabio_angeletti]

at 14:10 the event report hearbeat lost and this is the log:

Oct 15 10:14:19 zonedirector APMgr: Notify STAmgr for event Delete AP(60) on AP 00:00:00:00:00:00
Oct 15 10:14:19 zonedirector APMgr: Stop Fragment timer
Oct 15 10:14:19 zonedirector APMgr: Stop RF Statistics timer
Oct 15 10:14:19 zonedirector APMgr: Stop ConfigActivation timer
Oct 15 10:14:19 zonedirector APMgr: Stop Reset timer
Oct 15 10:14:19 zonedirector APMgr: Stop RogueAP_5G timer
Oct 15 10:14:19 zonedirector APMgr: Stop RogueAP_2.4G timer
Oct 15 10:14:19 zonedirector APMgr: Stop Statistics timer
Oct 15 10:14:19 zonedirector APMgr: Stop StaMgrWait timer
Oct 15 10:14:19 zonedirector APMgr: Stop ResponseWait timer
Oct 15 10:14:19 zonedirector APMgr: Stop Retransmit timer
Oct 15 10:14:19 zonedirector APMgr: Stop NeighborDead timer
Oct 15 10:14:19 zonedirector APMgr: Stop SessionID timer
Oct 15 10:14:19 zonedirector APMgr: Stop KeyLife timer
Oct 15 10:14:19 zonedirector APMgr: Stop HeartBeat timer
Oct 15 10:14:19 zonedirector APMgr: Stop Silent timer
Oct 15 10:14:19 zonedirector APMgr: Stop Discovery timer
Oct 15 10:14:19 zonedirector APMgr: Stop Idle timer
Oct 15 10:14:19 zonedirector APMgr: AP[00:00:00:00:00:00] State transfer from IDLE to RESET for Idle Timer Exp event
Oct 15 10:14:19 zonedirector APMgr: IPC receives from module APMgr for event Idle Timer Exp(0100002d) addr: 80.18.85.129(1032)
Oct 15 10:14:19 zonedirector APMgr: IPC sent to module APMgr for event Idle Timer Exp regarding 80.18.85.129(1032)
Oct 15 10:14:19 zonedirector APMgr: Timer Idle expired
Oct 15 10:14:18 zonedirector APMgr: Packet dropped for IDLE state.
Oct 15 10:14:18 zonedirector APMgr: Start Idle timer, timeout 5 seconds
Oct 15 10:14:18 zonedirector APMgr: Receive packet from 80.18.85.129(1031)
Oct 15 10:14:18 zonedirector APMgr: Decrypt Remote IPC(254) message from 80.18.85.129(12223) failed
Oct 15 10:14:18 zonedirector APMgr: Receive Remote IPC(254) message(seqnum 36) from 80.18.85.129(12223)
Oct 15 10:14:18 zonedirector APMgr: Receive packet from 80.18.85.129(12223)
Oct 15 10:14:18 zonedirector APMgr: Decrypt WTP Event Request(14) message from 80.18.85.129(12223) failed
Oct 15 10:14:18 zonedirector APMgr: Receive WTP Event Request(14) message(seqnum 35) from 80.18.85.129(12223)
Oct 15 10:14:18 zonedirector APMgr: Receive packet from 80.18.85.129(12223)
Oct 15 10:14:15 zonedirector APMgr: Decrypt Echo Request(22) message from 80.18.85.129(12223) failed
Oct 15 10:14:15 zonedirector APMgr: Receive Echo Request(22) message(seqnum 34) from 80.18.85.129(12223)
Oct 15 10:14:15 zonedirector APMgr: Receive packet from 80.18.85.129(12223)
Oct 15 10:14:14 zonedirector APMgr: Packet dropped for IDLE state.
Oct 15 10:14:14 zonedirector APMgr: Start Idle timer, timeout 5 seconds
Oct 15 10:14:14 zonedirector APMgr: create wtp_cb=0x577e40 for AP 80.18.85.129(1032)
Oct 15 10:14:14 zonedirector APMgr: Deleted malicious AP's list
Oct 15 10:14:14 zonedirector APMgr: Deleted malicious AP's list
Oct 15 10:14:14 zonedirector APMgr: Receive packet from 80.18.85.129(1032)
Oct 15 10:14:13 zonedirector APMgr: Packet dropped for IDLE state.
Oct 15 10:14:13 zonedirector APMgr: Start Idle timer, timeout 5 seconds
Oct 15 10:14:13 zonedirector APMgr: Receive packet from 80.18.85.129(1031)
Oct 15 10:14:12 zonedirector APMgr: Delete all pending requsts to AP 00:00:00:00:00:00
Oct 15 10:14:12 zonedirector APMgr: Stop Fragment timer
Oct 15 10:14:12 zonedirector APMgr: Stop RF Statistics timer
Oct 15 10:14:12 zonedirector APMgr: Stop ConfigActivation timer
Oct 15 10:14:12 zonedirector APMgr: Stop Reset timer
Oct 15 10:14:12 zonedirector APMgr: Stop RogueAP_5G timer
Oct 15 10:14:12 zonedirector APMgr: Stop RogueAP_2.4G timer
Oct 15 10:14:12 zonedirector APMgr: Stop Statistics timer
Oct 15 10:14:12 zonedirector APMgr: Stop StaMgrWait timer
Oct 15 10:14:12 zonedirector APMgr: Stop ResponseWait timer
Oct 15 10:14:12 zonedirector APMgr: Stop Retransmit timer
Oct 15 10:14:12 zonedirector APMgr: Stop NeighborDead timer
Oct 15 10:14:12 zonedirector APMgr: Stop SessionID timer
Oct 15 10:14:12 zonedirector APMgr: Stop KeyLife timer
Oct 15 10:14:12 zonedirector APMgr: Stop HeartBeat timer
Oct 15 10:14:12 zonedirector APMgr: Stop Silent timer
Oct 15 10:14:12 zonedirector APMgr: Stop Discovery timer
Oct 15 10:14:12 zonedirector APMgr: Stop Idle timer
Oct 15 10:14:12 zonedirector APMgr: Apmgr ready to free AP[00:00:00:00:00:00] wtp_cb=0x59fa80
Oct 15 10:14:12 zonedirector APMgr: AP[00:00:00:00:00:00] State transfer from RESET to RESET for Reset Timer Exp event
Oct 15 10:14:12 zonedirector APMgr: AP RESET retry count exhausted for AP:00:00:00:00:00:00 changing state to LWAPP_STATE_RESET_WTP
Oct 15 10:14:12 zonedirector APMgr: IPC receives from module APMgr for event Reset Timer Exp(01000045) addr: 80.18.85.129(1032)
Oct 15 10:14:12 zonedirector APMgr: IPC sent to module APMgr for event Reset Timer Exp regarding 80.18.85.129(1032)
Oct 15 10:14:12 zonedirector APMgr: Timer Reset expired
Oct 15 10:14:11 zonedirector stamgr: stamgr_AP_xml_house_keeping():complete updating ap cfg to emfd
Oct 15 10:14:11 zonedirector stamgr: _stamgr_esp_savecfg():update cfg to emfd complete
Oct 15 10:14:11 zonedirector stamgr: _stamgr_esp_savecfg():close espfd
Oct 15 10:14:11 zonedirector stamgr: _stamgr_esp_savecfg():used buffer for savecfg: 358
Oct 15 10:14:11 zonedirector stamgr: stamgr_esp_getajax():updobj ap-list
Oct 15 10:14:10 zonedirector APMgr: Decrypt Echo Request(22) message from 80.18.85.129(12223) failed
Oct 15 10:14:10 zonedirector APMgr: Receive Echo Request(22) message(seqnum 33) from 80.18.85.129(12223)
Oct 15 10:14:10 zonedirector APMgr: Receive packet from 80.18.85.129(12223)
Oct 15 10:14:10 zonedirector syslog: eventd_to_syslog():AP[apmed2@68:92:34:19:e7:b0] heartbeats lost
Oct 15 10:14:10 zonedirector APMgr: Stamgr requests to Del/Reset AP, but AP is not in run state. Just clean locally
Oct 15 10:14:10 zonedirector APMgr: Delete all pending requsts to AP 68:92:34:19:e7:b0
Oct 15 10:14:10 zonedirector APMgr: Stop Fragment timer
Oct 15 10:14:10 zonedirector APMgr: Stop RF Statistics timer
Oct 15 10:14:10 zonedirector APMgr: Stop ConfigActivation timer
Oct 15 10:14:10 zonedirector APMgr: Stop Reset timer
Oct 15 10:14:10 zonedirector APMgr: Stop RogueAP_5G timer
Oct 15 10:14:10 zonedirector APMgr: Stop RogueAP_2.4G timer
Oct 15 10:14:10 zonedirector APMgr: Stop Statistics timer
Oct 15 10:14:10 zonedirector APMgr: Stop StaMgrWait timer
Oct 15 10:14:10 zonedirector APMgr: Stop ResponseWait timer
Oct 15 10:14:10 zonedirector APMgr: Stop Retransmit timer
Oct 15 10:14:10 zonedirector APMgr: Stop NeighborDead timer
Oct 15 10:14:10 zonedirector APMgr: Stop SessionID timer
Oct 15 10:14:10 zonedirector APMgr: Stop KeyLife timer
Oct 15 10:14:10 zonedirector APMgr: Stop HeartBeat timer
Oct 15 10:14:10 zonedirector APMgr: Stop Silent timer
Oct 15 10:14:10 zonedirector APMgr: Stop Discovery timer
Oct 15 10:14:10 zonedirector APMgr: Stop Idle timer
Oct 15 10:14:10 zonedirector APMgr: Apmgr ready to free AP[68:92:34:19:e7:b0] wtp_cb=0x577e40
Oct 15 10:14:10 zonedirector stamgr: stamgr_handle_del_ap():AP 68:92:34:19:e7:b0 joined from L3 not found in AP hash table
Oct 15 10:14:10 zonedirector stamgr: stamgr_handle_del_ap():Receive an DEL-AP mssage, AP mac= 68:92:34:19:e7:b0
Oct 15 10:14:10 zonedirector stamgr: stamgr_ctrl_iface_receive():received msg from module 1, len = 196, id = 0x02000002
Oct 15 10:14:10 zonedirector APMgr: IPC sent to module Station Manager for event Del AP regarding 80.18.85.129(1029)
Oct 15 10:14:10 zonedirector APMgr: Notify STAmgr for event Delete AP(60) on AP 68:92:34:19:e7:b0
Oct 15 10:14:10 zonedirector APMgr: Cannot notify kernel for Delete AP 80.18.85.129(1029) event, -- No such file or directory(2)
Oct 15 10:14:10 zonedirector APMgr: command Reset AP rc code 2
Oct 15 10:14:10 zonedirector APMgr: STAmgr Reset AP(0x2000014) command for AP 68:92:34:19:e7:b0, state: RESET
Oct 15 10:14:10 zonedirector APMgr: IPC receives from module Station Manager for event Reset AP(02000014) addr: 80.18.85.129(1029)
Oct 15 10:14:10 zonedirector APMgr: Start Reset timer, timeout 3 seconds
Oct 15 10:14:10 zonedirector stamgr: stamgr_free_VAP(): id=245, addr=0xa644f8
Oct 15 10:14:10 zonedirector stamgr: stamgr_handle_del_ap():Receive an DEL-AP mssage, AP mac= 68:92:34:19:e7:b0
Oct 15 10:14:10 zonedirector stamgr: stamgr_ctrl_iface_receive():received msg from module 1, len = 196, id = 0x02000002
Oct 15 10:14:10 zonedirector APMgr: IPC sent to module Station Manager for event Del AP regarding 80.18.85.129(1029)
Oct 15 10:14:10 zonedirector APMgr: Notify STAmgr for event Delete AP(60) on AP 68:92:34:19:e7:b0
Oct 15 10:14:10 zonedirector APMgr: Stop Fragment timer
Oct 15 10:14:10 zonedirector APMgr: Stop RF Statistics timer
Oct 15 10:14:10 zonedirector APMgr: Stop ConfigActivation timer
Oct 15 10:14:10 zonedirector APMgr: Stop Reset timer
Oct 15 10:14:10 zonedirector APMgr: Stop RogueAP_5G timer
Oct 15 10:14:10 zonedirector APMgr: Stop RogueAP_2.4G timer
Oct 15 10:14:10 zonedirector APMgr: Stop Statistics timer
Oct 15 10:14:10 zonedirector APMgr: Stop StaMgrWait timer
Oct 15 10:14:10 zonedirector APMgr: Stop ResponseWait timer
Oct 15 10:14:10 zonedirector APMgr: Stop Retransmit timer
Oct 15 10:14:10 zonedirector APMgr: Stop NeighborDead timer
Oct 15 10:14:10 zonedirector APMgr: Stop SessionID timer
Oct 15 10:14:10 zonedirector APMgr: Stop KeyLife timer
Oct 15 10:14:10 zonedirector APMgr: Stop HeartBeat timer
Oct 15 10:14:10 zonedirector APMgr: Stop Silent timer
Oct 15 10:14:10 zonedirector APMgr: Stop Discovery timer
Oct 15 10:14:10 zonedirector APMgr: Stop Idle timer
Oct 15 10:14:10 zonedirector APMgr: AP[68:92:34:19:e7:b0] State transfer from RUN to RESET for HeartBeat Timer Exp event
Oct 15 10:14:10 zonedirector APMgr: IPC receives from module APMgr for event HeartBeat Timer Exp(01000030) addr: 80.18.85.129(1029)
Oct 15 10:14:10 zonedirector APMgr: IPC sent to module APMgr for event HeartBeat Timer Exp regarding 80.18.85.129(1029)
Oct 15 10:14:10 zonedirector APMgr: Timer HeartBeat expired
Oct 15 10:14:09 zonedirector APMgr: Packet dropped for RESET state.
Oct 15 10:14:09 zonedirector APMgr: Receive packet from 80.18.85.129(1032)
Oct 15 10:14:09 zonedirector APMgr: Start Reset timer, timeout 3 seconds
Oct 15 10:14:09 zonedirector stamgr: stamgr_handle_del_ap():AP 00:00:00:00:00:00 joined