ICX disconnected from Unleashed; connectivity issues between AP and ICX

[jacob_hutcheson]

Like everyone else here I have a Lennar-provided ICX 7150 and two APs. Something happened over the weekend and the ICX started showing up as Disconnected in the Unleashed GUI. Symptomatically things were 'connected', but traffic through the ICX to our ISP's stupid combo modem/router was having lots of issues.

I searched around on these forums (before finding the Lennar-specific section) and found some people with similar problems getting an ICX to connect to Unleashed. I tried factory resetting the ICX and re-adding it, but that didn't work. The ICX shows up in Unleashed as Pending, then prompts to enter the IP to join it (Auto Approve is enabled but seems like the ICX is in 'Router' mode per the docs?).

No matter what I have tried, after approving the ICX, it tries connecting and then shows up as Disconnected with a message that the ICX could not be pinged. 

I also updated the ICX to 08.0.90f via TFTP, but the result was the same.

I verified that LLDP is fine between the ICX and APs, but ICMP is not. I indeed can't ping the ICX from the AP, but I can ping it when I'm on the wired network from the ISP router. Interestingly when I'm connected to the router, I can't ping the AP directly. I can ping the AP if I'm on the wifi network being broadcast by Unleashed, but then I can't ping the ICX.

I know that was pretty hand-wavy but I can clarify if needed. 

[sergiosoto78]

Sorry 

what I meant was I could not see my switch either even after flashing the correct firmware and making sure it booted from the correct firmware.  The switch is not recognized by the unleashed network.  I have my R510s connected to the home network and those show up fine. 

the second part was for another thread

[nate_smith]

So I tried moving the switch over to battery backup to prevent it from going down in the future. Now it's failing to boot from primary in the logs and the secondary is the spr firmware. Anyone have any steps on what I should try next?

Jan  1 00:21:33:I:PKI: Trustpoint DEVICE_TRUSTPNT: Successfully loaded Manufacturing certificate file.   

Jan  1 00:21:33:I:System failed to boot from primary !!           

Jan  1 00:21:33:I:System: Cold start                              

Jan  1 00:21:32:D:DHCPC: starting dhcp client service on 656 port(s) 

Jan  1 00:21:32:I:Security: running-config was potentially changed by operator from console 

Edit: I got frustrated with it and flashed the latest switch firmware to both the primary and secondary using a usb and the newer manifest method. Had some issues with POE when it came back up, probably because I had the APs unplugged while I did the upgrade but they eventually got power and came back up.

[rob_raper]

I'm having this same problem too.   Things seemed to be working OK but noticed the switch is inaccessible from Unleashed. 

Tried logging in, the Lennar password no longer worked.  Tried lots of things.  USB C as console doesn't connect, computer sees no device on the other end at all.   Did a factory reset which allowed me to log in as sp-admin, but I end up in Router mode.   If I try to switch to the primary it doesn't boot, gives up after a while and goes back to Router.  (same "failed to boot form primary!") in log as seen above. 

Verified md5 of primary .. comes back correct, so the primary image is not corrupt.

Enterprise grade stuff should work better than this.   Terrible.  It's a good thing I don't really need to get in there to do anything. 

[rob_raper]

UPDATE: Following what nate_smith wrote, I tried updating to 8095d.   

SPS8095d was able to boot OK from primary.  This seems to to resolve whatever is causing SPS8090d not to boot.   Now that I have a switch image that safely boots, I copied it to secondary, so I will never be in router mode again.     Still remains unclear why primary/SPS8090d stopped being able to boot, even after a factory reset.  

BTW, the reason your POE stuff stopped working is because the newer versions contain a POE firmware update that gets applied after boot up.   You can see that flash progress in the log.   It took about 20 minutes for that to finish on mine.

I restarted the switch and it's remembering my logins and such now, so my problem seems resolved.   Let's just hope it sticks.   I'm hoping there was some bug fixed in this newer version.     I'm guessing most Lennar homeowners won't notice if their ICX falls into router mode, as the basic L2 switching + POE use case still works, the Unleashed app just complains that it can't connect to the switch. 

[nate_smith]

@rob_raper they’ll definitely notice if it’s in router mode if it’s hanging off another switch. That’s how I knew something was seriously wrong because all of my wireless devices weren’t communicating with my wired. At this point once Ubiquity has the 2.5GbE POE switches out of early access and they update their APs to 802.11ax and 2.5GbE ports, I’m replacing all the Ruckus equipment.